Jul 28 16:18:03.782369 ip-10-0-133-120 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 16:18:03.782383 ip-10-0-133-120 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 16:18:03.782394 ip-10-0-133-120 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 16:18:03.782732 ip-10-0-133-120 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 16:18:13.949157 ip-10-0-133-120 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 16:18:13.949176 ip-10-0-133-120 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a08e7349690646d1817e1efdb03916c7 --
Jul 28 16:20:14.481709 ip-10-0-133-120 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 16:20:14.894350 ip-10-0-133-120 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:20:14.894350 ip-10-0-133-120 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 16:20:14.894350 ip-10-0-133-120 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:20:14.894350 ip-10-0-133-120 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 16:20:14.894350 ip-10-0-133-120 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:20:14.895312 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.895219    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 16:20:14.900138 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900121    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:20:14.900138 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900137    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:20:14.900138 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900140    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900143    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900146    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900150    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900152    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900156    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900158    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900162    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900164    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900170    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900174    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900177    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900179    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900182    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900184    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900187    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900189    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900192    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900194    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900197    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:20:14.900228 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900200    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900202    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900207    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900209    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900212    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900215    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900218    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900220    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900223    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900225    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900228    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900231    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900233    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900236    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900239    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900242    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900250    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900253    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900255    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900258    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:20:14.900714 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900260    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900263    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900265    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900268    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900270    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900272    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900275    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900277    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900280    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900283    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900285    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900287    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900289    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900292    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900294    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900297    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900299    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900303    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900306    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900308    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:20:14.901179 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900310    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900313    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900316    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900318    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900320    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900323    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900325    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900328    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900331    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900340    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900343    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900346    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900348    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900350    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900353    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900357    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900360    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900362    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900365    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:20:14.901699 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900380    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900384    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900387    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900390    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900393    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900396    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900399    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900402    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900405    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900408    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900411    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900414    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900417    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900419    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900422    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900424    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900426    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.900429    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901312    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901324    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:20:14.902134 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901327    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901330    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901333    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901335    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901338    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901341    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901344    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901346    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901348    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901352    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901355    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901357    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901360    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901362    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901365    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901367    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901386    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901389    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901392    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901394    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:20:14.902624 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901398    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901401    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901403    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901406    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901408    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901411    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901414    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901416    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901418    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901421    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901423    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901426    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901429    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901432    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901435    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901437    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901440    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901443    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901445    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901448    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:20:14.903251 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901450    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901453    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901455    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901458    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901460    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901464    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901468    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901471    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901473    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901476    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901481    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901484    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901487    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901490    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901493    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901496    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901498    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901501    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901504    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901506    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:20:14.904009 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901508    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901512    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901514    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901517    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901519    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901522    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901524    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901527    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901530    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901533    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901535    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901538    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901541    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901543    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901546    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901548    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901551    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901553    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901556    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:20:14.904552 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901560    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901563    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901565    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901568    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901571    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901573    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901576    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901579    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901582    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901584    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901587    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901589    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901592    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901594    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901597    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901599    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901602    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901604    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.901996    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:20:14.905022 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902001    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902003    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902006    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902009    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902011    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902014    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902017    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902019    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902022    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902024    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902027    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902029    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902032    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902034    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902036    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902039    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902041    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902044    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902046    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902048    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:20:14.905494 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902052    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902054    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902057    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902059    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902062    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902064    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902066    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902069    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902072    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902074    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902077    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902079    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902081    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902084    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902087    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902089    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902091    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902094    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902097    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902099    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:20:14.905960 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902101    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902103    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902106    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902108    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902110    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902113    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902116    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902118    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902120    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902123    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902125    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902128    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902132    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902135    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902137    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902140    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902142    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902145    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902148    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:20:14.906483 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902153    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902157    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902160    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902163    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902165    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902168    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902171    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902173    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902176    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902179    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902181    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902184    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902186    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902189    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902191    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902194    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902196    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902199    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902201    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:20:14.906923 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902206    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902209    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902212    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902215    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902217    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902220    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902223    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902226    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902229    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902232    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902234    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902237    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902239    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902244    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902246    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902249    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902251    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902254    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902256    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.902258    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:20:14.907364 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902351    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902362    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902386    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902391    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902394    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902397    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902406    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902412    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902415    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902417    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902421    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902424    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902427    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902430    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902433    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902436    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902438    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902441    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902447    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902449    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902452    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902456    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902459    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902463    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 16:20:14.907851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902466    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902470    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902473    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902476    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902479    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902482    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902485    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902487    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902491    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902494    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902497    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902499    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902502    2573 flags.go:64] FLAG: --enable-server="true"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902505    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902509    2573 flags.go:64] FLAG: --event-burst="100"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902512    2573 flags.go:64] FLAG: --event-qps="50"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902515    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902518    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902521    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902524    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902527    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902530    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902533    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902535    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902538    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 16:20:14.908510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902541    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902546    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902549    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902551    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902554    2573 flags.go:64] FLAG: --feature-gates=""
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902558    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902561    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902564    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902568    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902571    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902574    2573 flags.go:64] FLAG: --help="false"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902576    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-133-120.ec2.internal"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902580    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902582    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902585    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902588    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902591    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902594    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902596    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902599    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902602    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902605    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902608    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902610    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 28 16:20:14.909109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902613    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902616    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902619    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902621    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902624    2573 flags.go:64] FLAG: --lock-file=""
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902627    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902630    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902633    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902638    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902643    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902645    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902648    2573 flags.go:64] FLAG: --logging-format="text"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902651    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902654    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902657    2573 flags.go:64] FLAG: --manifest-url=""
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902659    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902664    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902667    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902672    2573 flags.go:64] FLAG: --max-pods="110"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902674    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902677    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902680    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902683    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902686    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902689    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 16:20:14.909710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902692    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902699    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902702    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902705    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902708    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902711    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902716    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902719    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902721    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902724    2573 flags.go:64] FLAG: --port="10250"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902727    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902730    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b40813c4f2d58a3a"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902733    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902735    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902738    2573 flags.go:64] FLAG: --register-node="true"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902741    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902745    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902748    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902751    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902753    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902757    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902760    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902801    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902846    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902853    2573 flags.go:64] FLAG: --runonce="false"
Jul 28 16:20:14.910335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902858    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902864    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902870    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902875    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902881    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902887    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902892    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902902    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902907    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902912    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902916    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.902921    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903022    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903073    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903084    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903105    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903116    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903125    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903141    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903160    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903168    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903177    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903186    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903195    2573 flags.go:64] FLAG: --v="2"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903206    2573 flags.go:64] FLAG: --version="false"
Jul 28 16:20:14.910966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903216    2573 flags.go:64] FLAG: --vmodule=""
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903226    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.903241    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904319    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904330    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904335    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904339    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904344    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904349    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904353    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904357    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904360    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904364    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904368    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904388    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904393    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904397    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904401    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904405    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904409    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:20:14.911583 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904418    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904423    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904426    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904430    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904434    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904438    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904442    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904446    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904449    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904454    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904457    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904461    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904464    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904468    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904473    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904476    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904478    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904481    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904484    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:20:14.912053 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904486    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904489    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904492    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904495    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904497    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904500    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904503    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904505    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904508    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904511    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904513    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904517    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904519    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904522    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904525    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904527    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904530    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904532    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904535    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904537    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:20:14.912515 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904540    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904542    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904545    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904547    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904550    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904552    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904555    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904559    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904563    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904567    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904569    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904572    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904582    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904585    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904588    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904591    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904593    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904596    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904599    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904601    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:20:14.913015 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904604    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904607    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904611    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904614    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904617    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904621    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904625    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904627    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904630    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904633    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904635    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904638    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904641    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904643    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904646    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904649    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904651    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904653    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904656    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:20:14.913535 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904659    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:20:14.913977 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904662    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:20:14.913977 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904664    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:20:14.913977 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.904667    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:20:14.913977 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.905208    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:20:14.914688 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.914669    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 16:20:14.914726 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.914688    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 16:20:14.915640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.915631    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 16:20:14.915671 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.915645    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 16:20:14.915712 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915704    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:20:14.915712 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915711    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915715    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915718    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915720    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915723    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915725    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915728    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915731    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915734    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915736    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915739    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915742    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915746    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915748    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915751    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915754    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915756    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915758    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915761    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915763    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:20:14.915766 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915766    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915769    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915772    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915774    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915777    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915780    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915783    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915785    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915788    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915790    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915793    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915795    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915798    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915801    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915804    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915806    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915809    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915811    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915813    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915816    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:20:14.916239 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915818    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915820    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915825    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915828    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915831    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915835    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915838    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915841    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915844    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915846    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915849    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915851    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915853    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915856    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915859    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915862    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915864    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915867    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915869    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:20:14.916738 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915872    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915874    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915877    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915879    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915882    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915884    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915887    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915889    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915891    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915895    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915898    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915901    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915904    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915906    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915908    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915911    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915913    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915916    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915919    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:20:14.917223 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915921    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915925    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915928    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915930    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915933    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915935    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915937    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915940    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915943    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915946    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915948    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915951    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915953    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915955    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915958    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915960    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915962    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915965    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915967    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:20:14.917677 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.915970    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.915975    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916085    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916089    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916091    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916095    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916097    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916100    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916103    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916105    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916107    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916110    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916112    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916115    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916118    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:20:14.918128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916120    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916124    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916127    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916130    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916133    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916135    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916138    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916141    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916144    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916147    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916150    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916153    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916155    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916158    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916160    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916163    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916166    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916169    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916172    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:20:14.918638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916175    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916177    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916180    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916182    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916184    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916187    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916189    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916192    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916195    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916197    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916200    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916202    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916204    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916207    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916210    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916212    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916215    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916218    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916220    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916223    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:20:14.919098 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916225    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916228    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916230    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916233    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916235    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916238    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916240    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916243    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916245    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916248    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916250    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916253    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916255    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916258    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916260    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916263    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916265    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916267    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916269    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916272    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:20:14.919598 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916275    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916277    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916279    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916282    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916284    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916288    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916292    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916294    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916297    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916299    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916302    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916304    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916307    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916309    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916312    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916314    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916316    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916319    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916321    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916323    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:20:14.920081 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916326    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916328    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916331    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916333    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916336    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916338    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:14.916340    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.916345    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.916468    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.918513    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.919699    2573 server.go:1019] "Starting client certificate rotation"
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.919797    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:20:14.920615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.920561    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:20:14.922290 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.922277    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:20:14.925038 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.925022    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:20:14.933386 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.933360    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 28 16:20:14.935410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.935393    2573 log.go:25] "Validated CRI v1 image API"
Jul 28 16:20:14.936399 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.936357    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 16:20:14.938930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.938908    2573 fs.go:135] Filesystem UUIDs: map[776e1629-64f4-4a98-b2c5-a9e31cdec0ba:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ccd195d6-587d-4cbd-9fdb-908a47482648:/dev/nvme0n1p4]
Jul 28 16:20:14.938988 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.938931    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 16:20:14.945845 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.945721    2573 manager.go:216] Machine: {Timestamp:2026-07-28 16:20:14.944458124 +0000 UTC m=+0.337774653 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3202166 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27ad0f6c05f305fe43c966f4c54d7f SystemUUID:ec27ad0f-6c05-f305-fe43-c966f4c54d7f BootID:a08e7349-6906-46d1-817e-1efdb03916c7 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:8f:a2:2b:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:8f:a2:2b:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:b4:ad:58:3e:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 16:20:14.945845 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.945833    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 16:20:14.946018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.946001    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 16:20:14.947528 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.947504    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:20:14.949189 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.949163    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 16:20:14.949351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.949195    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 16:20:14.950940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.950927    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 16:20:14.951002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.950946    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 16:20:14.951002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.950956    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:20:14.951593 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.951580    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:20:14.952440 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.952429    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 16:20:14.954891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.954876    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:20:14.955034 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955023    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 16:20:14.955089 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955061    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 16:20:14.955089 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955074    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 16:20:14.955181 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955092    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 16:20:14.955181 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955106    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 28 16:20:14.955181 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955120    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 16:20:14.955536 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955521    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:20:14.955601 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.955549    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:20:14.957827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.957809    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 16:20:14.965106 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.965090    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 16:20:14.965181 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.965120    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 16:20:14.965983 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.965957    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 16:20:14.966086 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.966068    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 16:20:14.966729 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966714    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 16:20:14.966782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966738    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 16:20:14.966782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966747    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 16:20:14.966782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966756    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 16:20:14.966782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966768    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 16:20:14.966782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966775    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 16:20:14.966782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966786    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 16:20:14.966941 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966793    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 16:20:14.966941 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966800    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 16:20:14.966941 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966806    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 16:20:14.966941 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966814    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 16:20:14.966941 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.966820    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 16:20:14.968707 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.968691    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 16:20:14.968707 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.968705    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 16:20:14.970841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.970824    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 16:20:14.971831 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.971818    2573 server.go:1268] "Started kubelet"
Jul 28 16:20:14.971948 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.971911    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 16:20:14.972103 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.972061    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 16:20:14.972246 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.972227    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 16:20:14.972818 ip-10-0-133-120 systemd[1]: Started Kubernetes Kubelet.
Jul 28 16:20:14.973871 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.973857    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 16:20:14.973934 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.973920    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 16:20:14.977305 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.977287    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 16:20:14.980621 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.980606    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 16:20:14.980621 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.980619    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 16:20:14.980781 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.980764    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97xfv"
Jul 28 16:20:14.981126 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.981093    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:14.981227 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981209    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 16:20:14.981306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981236    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 16:20:14.981306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981247    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 16:20:14.981422 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981321    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 16:20:14.981422 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981331    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 16:20:14.981847 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981832    2573 factory.go:153] Registering CRI-O factory
Jul 28 16:20:14.981940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981852    2573 factory.go:223] Registration of the crio container factory successfully
Jul 28 16:20:14.981940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981902    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 16:20:14.981940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981910    2573 factory.go:55] Registering systemd factory
Jul 28 16:20:14.981940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981915    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 28 16:20:14.981940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981931    2573 factory.go:103] Registering Raw factory
Jul 28 16:20:14.981940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.981942    2573 manager.go:1196] Started watching for new ooms in manager
Jul 28 16:20:14.983000 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.982624    2573 manager.go:349] Starting recovery of all containers
Jul 28 16:20:14.985777 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.985745    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 16:20:14.985906 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.985875    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 16:20:14.986712 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:14.982747    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-120.ec2.internal.18c68085ebaa4bac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-120.ec2.internal,UID:ip-10-0-133-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-120.ec2.internal,},FirstTimestamp:2026-07-28 16:20:14.9717923 +0000 UTC m=+0.365108831,LastTimestamp:2026-07-28 16:20:14.9717923 +0000 UTC m=+0.365108831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-120.ec2.internal,}"
Jul 28 16:20:14.986827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.986724    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97xfv"
Jul 28 16:20:14.996198 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:14.996180    2573 manager.go:354] Recovery completed
Jul 28 16:20:15.000326 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.000314    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.002398 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.002366    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.002470 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.002412    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.002470 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.002426    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.003204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.003180    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 16:20:15.003204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.003197    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 16:20:15.003337 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.003215    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:20:15.003715 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.003684    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.004807 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.004792    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.004892 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.004824    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.004892 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.004840    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.005038 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.004932    2573 policy_none.go:49] "None policy: Start"
Jul 28 16:20:15.005038 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.004945    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 16:20:15.005038 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.004955    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 16:20:15.006086 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.006074    2573 policy_none.go:47] "Start"
Jul 28 16:20:15.038010 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.037989    2573 manager.go:337] "Starting Device Plugin manager"
Jul 28 16:20:15.038104 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.038028    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 16:20:15.038104 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.038038    2573 server.go:87] "Starting device plugin registration server"
Jul 28 16:20:15.038288 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.038277    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 16:20:15.038322 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.038293    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 16:20:15.038445 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.038425    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 16:20:15.038547 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.038502    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 16:20:15.038547 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.038510    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 16:20:15.039138 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.039114    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 16:20:15.039219 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.039175    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.138812 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.138774    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.140112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.139802    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.140112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.139829    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.140112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.139839    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.140112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.139866    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.146682 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.146614    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.146682 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.146641    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-120.ec2.internal\": node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.162860 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.162828    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.165558 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.165527    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 16:20:15.167058 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.167040    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 16:20:15.167119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.167087    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 16:20:15.167119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.167113    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 16:20:15.167198 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.167151    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 16:20:15.169778 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.169761    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:20:15.263236 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.263193    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.267266 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.267243    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"]
Jul 28 16:20:15.267330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.267316    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.268901 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.268885    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.268991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.268915    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.268991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.268924    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.270168 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.270155    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.270302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.270282    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.270726 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.270712    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.270869 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.270857    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.270919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.270882    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.270919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.270891    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.271414 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.271401    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.271462 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.271428    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.271462 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.271439    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.272056 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.272043    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.272088 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.272069    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:20:15.272695 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.272679    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:20:15.272770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.272705    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:20:15.272770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.272718    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:20:15.282981 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.282960    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d136da5e3ede6720eaff51fc218626d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"9d136da5e3ede6720eaff51fc218626d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.283077 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.282987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a87de93e34c8837496bc8bb7c2576ca1-config\") pod \"kube-apiserver-proxy-ip-10-0-133-120.ec2.internal\" (UID: \"a87de93e34c8837496bc8bb7c2576ca1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.283077 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.283004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d136da5e3ede6720eaff51fc218626d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"9d136da5e3ede6720eaff51fc218626d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.297242 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.297220    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-120.ec2.internal\" not found" node="ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.301613 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.301594    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-120.ec2.internal\" not found" node="ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.363883 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.363850    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.383351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.383325    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d136da5e3ede6720eaff51fc218626d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"9d136da5e3ede6720eaff51fc218626d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.383463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.383356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a87de93e34c8837496bc8bb7c2576ca1-config\") pod \"kube-apiserver-proxy-ip-10-0-133-120.ec2.internal\" (UID: \"a87de93e34c8837496bc8bb7c2576ca1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.383463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.383395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d136da5e3ede6720eaff51fc218626d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"9d136da5e3ede6720eaff51fc218626d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.383463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.383422    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d136da5e3ede6720eaff51fc218626d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"9d136da5e3ede6720eaff51fc218626d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.383463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.383433    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d136da5e3ede6720eaff51fc218626d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal\" (UID: \"9d136da5e3ede6720eaff51fc218626d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.383463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.383436    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a87de93e34c8837496bc8bb7c2576ca1-config\") pod \"kube-apiserver-proxy-ip-10-0-133-120.ec2.internal\" (UID: \"a87de93e34c8837496bc8bb7c2576ca1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.464050 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.463975    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.564542 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.564504    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.599998 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.599968    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.605173 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.605154    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:15.665794 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.665590    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.766519 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.766436    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.867018 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.866976    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.918902 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.918871    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 16:20:15.919352 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.919012    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:20:15.967393 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:15.967350    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:15.991649 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.991610    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 16:15:15 +0000 UTC" deadline="2028-03-05 22:37:42.252882415 +0000 UTC"
Jul 28 16:20:15.991649 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:15.991641    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14070h17m26.261244214s"
Jul 28 16:20:16.067944 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:16.067911    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:16.142140 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:16.142088    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d136da5e3ede6720eaff51fc218626d.slice/crio-b6e0a7587f45a9252f50930133c050b68fd8c1de2d79270bc07d96c88c70b727 WatchSource:0}: Error finding container b6e0a7587f45a9252f50930133c050b68fd8c1de2d79270bc07d96c88c70b727: Status 404 returned error can't find the container with id b6e0a7587f45a9252f50930133c050b68fd8c1de2d79270bc07d96c88c70b727
Jul 28 16:20:16.147655 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.147636    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:20:16.170122 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:16.170051    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-120.ec2.internal\" not found"
Jul 28 16:20:16.172415 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.172358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" event={"ID":"9d136da5e3ede6720eaff51fc218626d","Type":"ContainerStarted","Data":"b6e0a7587f45a9252f50930133c050b68fd8c1de2d79270bc07d96c88c70b727"}
Jul 28 16:20:16.193880 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:16.193858    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87de93e34c8837496bc8bb7c2576ca1.slice/crio-2264bd50764e404274a3710992b55d0b17ca079d93acef9eb938a4cf2038abd9 WatchSource:0}: Error finding container 2264bd50764e404274a3710992b55d0b17ca079d93acef9eb938a4cf2038abd9: Status 404 returned error can't find the container with id 2264bd50764e404274a3710992b55d0b17ca079d93acef9eb938a4cf2038abd9
Jul 28 16:20:16.207308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.207286    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:20:16.268395 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.268366    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:20:16.281733 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.281706    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:16.298077 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.298020    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:20:16.299039 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.299025    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal"
Jul 28 16:20:16.306879 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.306855    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:20:16.541416 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.541372    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:20:16.955856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.955522    2573 apiserver.go:52] "Watching apiserver"
Jul 28 16:20:16.961728 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.961695    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 16:20:16.962310 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.962260    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal","openshift-multus/network-metrics-daemon-nfzgs","openshift-network-diagnostics/network-check-target-6h5n9","openshift-network-operator/iptables-alerter-xcnfm","openshift-ovn-kubernetes/ovnkube-node-pszj4","kube-system/global-pull-secret-syncer-zx225","kube-system/konnectivity-agent-wnvwk","kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal","openshift-cluster-node-tuning-operator/tuned-75vvp","openshift-dns/node-resolver-wwffh","openshift-image-registry/node-ca-vm6qz","openshift-multus/multus-additional-cni-plugins-6hfsv","openshift-multus/multus-mf4gt"]
Jul 28 16:20:16.964018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.963992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:16.964157 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:16.964083    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:16.965259 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.965236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:16.965371 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:16.965303    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:16.967107 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.967015    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:16.967107 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:16.967074    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:16.968624 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.968565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:16.970204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.970178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.971683 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.971661    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.973824 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.973803    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 16:20:16.974319 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.974302    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.975783 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.975761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.975877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.974495    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pjrq4\""
Jul 28 16:20:16.977533 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.977508    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.977638 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.977551    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2dx7w\""
Jul 28 16:20:16.977700 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.977669    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 16:20:16.977758 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.977725    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.977852 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.977830    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 16:20:16.977985 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.977874    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 16:20:16.978132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.978113    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 16:20:16.978443 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.978423    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:16.978720 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.978701    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.978911 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.978894    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.979089 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.979073    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 16:20:16.979652 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.979633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.981309 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.980478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:16.982139 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.982067    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 16:20:16.984951 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.984928    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g6ptl\""
Jul 28 16:20:16.985189 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.985168    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:16.987571 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.987549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:16.988071 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988048    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.988513 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988318    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 16:20:16.988513 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988435    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.988716 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988693    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.988818 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988740    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 16:20:16.988818 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988647    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.989026 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.988595    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wj5md\""
Jul 28 16:20:16.989122 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989103    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 16:20:16.989180 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989146    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zwzj7\""
Jul 28 16:20:16.989394 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989366    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gkr2c\""
Jul 28 16:20:16.989481 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989470    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhmz\""
Jul 28 16:20:16.989757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989737    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.989886 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989764    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.989961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.989930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:16.991493 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.991471    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 16:20:16.991733 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.991711    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 16:20:16.991894 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.991877    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 16:20:16.992221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.992195    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 16:20:16.992479 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.992443    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 16:20:16.992853 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.992788    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-469jb\""
Jul 28 16:20:16.993325 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07640db5-ab1c-4797-870b-4b9f3767dbe1-agent-certs\") pod \"konnectivity-agent-wnvwk\" (UID: \"07640db5-ab1c-4797-870b-4b9f3767dbe1\") " pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:16.993408 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07640db5-ab1c-4797-870b-4b9f3767dbe1-konnectivity-ca\") pod \"konnectivity-agent-wnvwk\" (UID: \"07640db5-ab1c-4797-870b-4b9f3767dbe1\") " pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:16.993408 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-host\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993408 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-serviceca\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-run-netns\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993441    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovnkube-script-lib\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysctl-d\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysctl-conf\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993534    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-sys\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rpj\" (UniqueName: \"kubernetes.io/projected/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-kube-api-access-f8rpj\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:16.993590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-run\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993633    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-slash\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-var-lib-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-systemd\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-lib-modules\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-var-lib-kubelet\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-registration-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.993876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-tuned\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1bca669-4acf-49c5-8686-5928990e6e9f-hosts-file\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovnkube-config\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993970    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-socket-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.993994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwsc\" (UniqueName: \"kubernetes.io/projected/9deb4699-aee3-4e43-a1fa-0560f6328774-kube-api-access-ngwsc\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-tmp\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994042    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-log-socket\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-cni-bin\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysconfig\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994119    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-kubelet-config\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-dbus\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1bca669-4acf-49c5-8686-5928990e6e9f-tmp-dir\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-kubelet\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-ovn\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9j8\" (UniqueName: \"kubernetes.io/projected/c1bca669-4acf-49c5-8686-5928990e6e9f-kube-api-access-nb9j8\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994266    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-iptables-alerter-script\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-systemd-units\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994315    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-env-overrides\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcvk\" (UniqueName: \"kubernetes.io/projected/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-kube-api-access-kfcvk\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-host\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x6z\" (UniqueName: \"kubernetes.io/projected/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-kube-api-access-z8x6z\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-run-ovn-kubernetes\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994499    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-etc-selinux\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994537    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-sys-fs\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-device-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-modprobe-d\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zdg\" (UniqueName: \"kubernetes.io/projected/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-kube-api-access-h2zdg\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:16.994862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-systemd\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-cni-netd\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovn-node-metrics-cert\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994767    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-kubernetes\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994793    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-host-slash\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-etc-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-node-log\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:16.995640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.994875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7klq\" (UniqueName: \"kubernetes.io/projected/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-kube-api-access-b7klq\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:16.999100 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.999077    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 16:20:16.999381 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:16.999309    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l45qj\""
Jul 28 16:20:17.008302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.008279    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:20:17.032623 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.032593    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-plktc"
Jul 28 16:20:17.041642 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.041618    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-plktc"
Jul 28 16:20:17.082565 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.082530    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 16:20:17.095355 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-modprobe-d\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.095355 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095364    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zdg\" (UniqueName: \"kubernetes.io/projected/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-kube-api-access-h2zdg\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.095617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.095617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-kubelet\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.095617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095531    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-modprobe-d\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.095617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:17.095617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-systemd\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.095617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095603    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-cni-netd\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovn-node-metrics-cert\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-kubernetes\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-system-cni-dir\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-socket-dir-parent\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095703    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-systemd\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-conf-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-host-slash\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-etc-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-node-log\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7klq\" (UniqueName: \"kubernetes.io/projected/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-kube-api-access-b7klq\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07640db5-ab1c-4797-870b-4b9f3767dbe1-agent-certs\") pod \"konnectivity-agent-wnvwk\" (UID: \"07640db5-ab1c-4797-870b-4b9f3767dbe1\") " pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07640db5-ab1c-4797-870b-4b9f3767dbe1-konnectivity-ca\") pod \"konnectivity-agent-wnvwk\" (UID: \"07640db5-ab1c-4797-870b-4b9f3767dbe1\") " pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095887    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-kubernetes\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095904    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-hostroot\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.095928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-host\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-serviceca\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095991    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-etc-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096033    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-node-log\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-netns\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096073    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-cni-multus\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-run-netns\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096127    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovnkube-script-lib\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysctl-d\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysctl-conf\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-sys\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rpj\" (UniqueName: \"kubernetes.io/projected/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-kube-api-access-f8rpj\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8knz\" (UniqueName: \"kubernetes.io/projected/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-kube-api-access-v8knz\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-run\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096355    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:17.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-slash\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-var-lib-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-systemd\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-lib-modules\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-var-lib-kubelet\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096542    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cnibin\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-os-release\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-registration-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-tuned\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1bca669-4acf-49c5-8686-5928990e6e9f-hosts-file\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovnkube-config\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-socket-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwsc\" (UniqueName: \"kubernetes.io/projected/9deb4699-aee3-4e43-a1fa-0560f6328774-kube-api-access-ngwsc\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-tmp\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.097520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-system-cni-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-serviceca\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-etc-kubernetes\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-host-slash\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-log-socket\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-cni-bin\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-run-netns\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysconfig\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.096995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-kubelet-config\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-dbus\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097082    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-os-release\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-cni-bin\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1bca669-4acf-49c5-8686-5928990e6e9f-tmp-dir\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-kubelet\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-ovn\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-cni-binary-copy\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.098351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-k8s-cni-cncf-io\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9j8\" (UniqueName: \"kubernetes.io/projected/c1bca669-4acf-49c5-8686-5928990e6e9f-kube-api-access-nb9j8\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-iptables-alerter-script\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-systemd-units\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-env-overrides\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovnkube-script-lib\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcvk\" (UniqueName: \"kubernetes.io/projected/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-kube-api-access-kfcvk\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-host\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x6z\" (UniqueName: \"kubernetes.io/projected/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-kube-api-access-z8x6z\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097506    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-run-ovn-kubernetes\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-etc-selinux\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-sys-fs\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097609    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysctl-conf\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097614    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnwl\" (UniqueName: \"kubernetes.io/projected/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-kube-api-access-9pnwl\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097640    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-var-lib-kubelet\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.099211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097644    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-cnibin\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.095753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-cni-netd\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-device-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-registration-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-multus-certs\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-kubelet-config\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-cni-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysconfig\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-sys\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-host\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097895    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-run\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097901    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-dbus\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-slash\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097994    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-var-lib-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098050    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-openvswitch\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098088    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-lib-modules\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-systemd\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098135    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-device-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.100097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-daemon-config\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098386    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1bca669-4acf-49c5-8686-5928990e6e9f-tmp-dir\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1bca669-4acf-49c5-8686-5928990e6e9f-hosts-file\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-socket-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-log-socket\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-cni-bin\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.098973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovnkube-config\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-kubelet\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.099066    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-run-ovn\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.099146    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:20:17.599109906 +0000 UTC m=+2.992426426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099379    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-run-ovn-kubernetes\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099461    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-systemd-units\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099818    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-iptables-alerter-script\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099841    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-env-overrides\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099918    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-sys-fs\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.099933    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-host\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.100961 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.100023    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:17.102105 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.100073    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:17.600055001 +0000 UTC m=+2.993371520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:17.102105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.100205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.102105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.097508    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-sysctl-d\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.102105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.100355    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9deb4699-aee3-4e43-a1fa-0560f6328774-etc-selinux\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.102105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.101053    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07640db5-ab1c-4797-870b-4b9f3767dbe1-konnectivity-ca\") pod \"konnectivity-agent-wnvwk\" (UID: \"07640db5-ab1c-4797-870b-4b9f3767dbe1\") " pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:17.102105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.101145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-etc-tuned\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.102508 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.102360    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-ovn-node-metrics-cert\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.103361 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.103334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-tmp\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.103489 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.103474    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:17.103556 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.103497    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:17.103556 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.103512    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:17.103667 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.103582    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:17.60356463 +0000 UTC m=+2.996881164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:17.105226 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.104689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07640db5-ab1c-4797-870b-4b9f3767dbe1-agent-certs\") pod \"konnectivity-agent-wnvwk\" (UID: \"07640db5-ab1c-4797-870b-4b9f3767dbe1\") " pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:17.107683 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.105950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zdg\" (UniqueName: \"kubernetes.io/projected/5785ce30-a7d4-4670-ba3a-fb7b162a1fec-kube-api-access-h2zdg\") pod \"tuned-75vvp\" (UID: \"5785ce30-a7d4-4670-ba3a-fb7b162a1fec\") " pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.107683 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.106823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7klq\" (UniqueName: \"kubernetes.io/projected/19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53-kube-api-access-b7klq\") pod \"iptables-alerter-xcnfm\" (UID: \"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53\") " pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.114002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.113976    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwsc\" (UniqueName: \"kubernetes.io/projected/9deb4699-aee3-4e43-a1fa-0560f6328774-kube-api-access-ngwsc\") pod \"aws-ebs-csi-driver-node-4j9tc\" (UID: \"9deb4699-aee3-4e43-a1fa-0560f6328774\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.114623 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.114603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rpj\" (UniqueName: \"kubernetes.io/projected/b7a9e889-bb8a-4c9d-8fe0-12f2200c179c-kube-api-access-f8rpj\") pod \"node-ca-vm6qz\" (UID: \"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c\") " pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.117838 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.117792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9j8\" (UniqueName: \"kubernetes.io/projected/c1bca669-4acf-49c5-8686-5928990e6e9f-kube-api-access-nb9j8\") pod \"node-resolver-wwffh\" (UID: \"c1bca669-4acf-49c5-8686-5928990e6e9f\") " pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.117923 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.117845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x6z\" (UniqueName: \"kubernetes.io/projected/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-kube-api-access-z8x6z\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:17.118164 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.118145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcvk\" (UniqueName: \"kubernetes.io/projected/0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97-kube-api-access-kfcvk\") pod \"ovnkube-node-pszj4\" (UID: \"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97\") " pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.175960 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.175916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" event={"ID":"a87de93e34c8837496bc8bb7c2576ca1","Type":"ContainerStarted","Data":"2264bd50764e404274a3710992b55d0b17ca079d93acef9eb938a4cf2038abd9"}
Jul 28 16:20:17.198808 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cnibin\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.198808 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-os-release\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-system-cni-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198876    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cnibin\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-system-cni-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-os-release\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-etc-kubernetes\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.198997    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-os-release\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199026    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-etc-kubernetes\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-cni-bin\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-cni-binary-copy\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199121    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-cni-bin\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-k8s-cni-cncf-io\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-os-release\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-k8s-cni-cncf-io\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnwl\" (UniqueName: \"kubernetes.io/projected/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-kube-api-access-9pnwl\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-cnibin\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-multus-certs\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-cni-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199358    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-daemon-config\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-kubelet\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.199478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-system-cni-dir\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-socket-dir-parent\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-conf-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-hostroot\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-netns\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-cni-multus\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8knz\" (UniqueName: \"kubernetes.io/projected/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-kube-api-access-v8knz\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199741    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-cni-binary-copy\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-multus-certs\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199886    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-cni-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-cnibin\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-conf-dir\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-hostroot\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.199980    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-kubelet\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.200014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-run-netns\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200967 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.200029    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-host-var-lib-cni-multus\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200967 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.200069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-system-cni-dir\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.200967 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.200075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-socket-dir-parent\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200967 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.200385    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-multus-daemon-config\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.200967 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.200485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.208900 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.208835    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnwl\" (UniqueName: \"kubernetes.io/projected/ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9-kube-api-access-9pnwl\") pod \"multus-additional-cni-plugins-6hfsv\" (UID: \"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9\") " pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.209024 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.208969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8knz\" (UniqueName: \"kubernetes.io/projected/34b01b86-43a7-4891-aa95-8bd3ce6a0bf6-kube-api-access-v8knz\") pod \"multus-mf4gt\" (UID: \"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6\") " pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.217612 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.217587    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:20:17.296739 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.296691    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xcnfm"
Jul 28 16:20:17.303862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.303837    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:17.314044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.314020    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc"
Jul 28 16:20:17.321106 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.321080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:17.336765 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.336734    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwffh"
Jul 28 16:20:17.345678 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.345654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75vvp"
Jul 28 16:20:17.353704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.353681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vm6qz"
Jul 28 16:20:17.360411 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.360388    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hfsv"
Jul 28 16:20:17.367324 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.367306    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mf4gt"
Jul 28 16:20:17.602438 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.602331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:17.602614 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.602432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:17.602614 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.602528    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:17.602614 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.602548    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:17.602614 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.602613    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:18.602589573 +0000 UTC m=+3.995906103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:17.602794 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.602635    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:20:18.602624437 +0000 UTC m=+3.995940960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:17.703368 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:17.703335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:17.703559 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.703510    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:17.703559 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.703533    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:17.703559 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.703544    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:17.703693 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:17.703609    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:18.70358855 +0000 UTC m=+4.096905096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:18.044706 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.044639    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 16:15:17 +0000 UTC" deadline="2028-01-31 12:50:43.210609248 +0000 UTC"
Jul 28 16:20:18.044706 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.044692    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13244h30m25.16591933s"
Jul 28 16:20:18.136413 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:18.136375    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3b52ec_8aa9_44bc_aa2f_14eb3a998ba9.slice/crio-8ee6f7674b13f2e94be9d7338e1cae1a748a60be8fa664b19ba8ac0985e9d283 WatchSource:0}: Error finding container 8ee6f7674b13f2e94be9d7338e1cae1a748a60be8fa664b19ba8ac0985e9d283: Status 404 returned error can't find the container with id 8ee6f7674b13f2e94be9d7338e1cae1a748a60be8fa664b19ba8ac0985e9d283
Jul 28 16:20:18.168388 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.167937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:18.168388 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.168061    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:18.168388 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.168143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:18.168388 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.168223    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:18.180618 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.180593    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75vvp" event={"ID":"5785ce30-a7d4-4670-ba3a-fb7b162a1fec","Type":"ContainerStarted","Data":"69a801b175f5ea1de7052104a24bbd74841019112a7f41faf47968e48c6cb906"}
Jul 28 16:20:18.183584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.183550    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwffh" event={"ID":"c1bca669-4acf-49c5-8686-5928990e6e9f","Type":"ContainerStarted","Data":"f0f31e49c47921264904590c2d0c7a4e8ea693b41839602545616923ee05d186"}
Jul 28 16:20:18.186200 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.186174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mf4gt" event={"ID":"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6","Type":"ContainerStarted","Data":"9836b086a0da673ab0c9b8bd6a5487cbe5c74a83d618cd67c38acee444b28ce7"}
Jul 28 16:20:18.187395 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.187349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnvwk" event={"ID":"07640db5-ab1c-4797-870b-4b9f3767dbe1","Type":"ContainerStarted","Data":"f3d03174188eb7dd2d6443507e00c9535d983362e7172c60e7d7eee1d7fec165"}
Jul 28 16:20:18.188530 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.188511    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc" event={"ID":"9deb4699-aee3-4e43-a1fa-0560f6328774","Type":"ContainerStarted","Data":"9fd3f11c59d76e5e5b3f3a53407480589b84a71d6a05eb92fb40e70e3ac8ec36"}
Jul 28 16:20:18.189860 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.189772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm6qz" event={"ID":"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c","Type":"ContainerStarted","Data":"f585f1a3aaf8d30e5a6d60e2f2c97361ba73c4d4736fb6ad58dc75a7fe5a8786"}
Jul 28 16:20:18.191013 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.190985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xcnfm" event={"ID":"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53","Type":"ContainerStarted","Data":"5d82b82fe65b215319d546bf6aad1f01075bcc29b72f7603875678066c8e66f7"}
Jul 28 16:20:18.192172 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.192149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"9fc20e5fd77e09cf12f327e4bfe34136b1635defd9ea6fdb1590a8e4e461e658"}
Jul 28 16:20:18.193200 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.193182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerStarted","Data":"8ee6f7674b13f2e94be9d7338e1cae1a748a60be8fa664b19ba8ac0985e9d283"}
Jul 28 16:20:18.613045 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.612670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:18.613045 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.613000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:18.613276 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.612908    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:18.613276 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.613239    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:20:20.613219073 +0000 UTC m=+6.006535591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:18.613623 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.613170    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:18.613756 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.613665    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:20.613647158 +0000 UTC m=+6.006963680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:18.714439 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:18.713717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:18.714439 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.713974    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:18.714439 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.714000    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:18.714439 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.714014    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:18.714439 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:18.714075    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:20.714057076 +0000 UTC m=+6.107373598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:19.045246 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:19.045043    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 16:15:17 +0000 UTC" deadline="2028-01-22 02:38:38.103178758 +0000 UTC"
Jul 28 16:20:19.045246 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:19.045109    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13018h18m19.058074049s"
Jul 28 16:20:19.170601 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:19.170020    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:19.170601 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:19.170148    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:19.205717 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:19.205678    2573 generic.go:356] "Generic (PLEG): container finished" podID="9d136da5e3ede6720eaff51fc218626d" containerID="631975076ad168af31b1902e088165d927950ca6b1ed97ebb4da7be8615722bd" exitCode=0
Jul 28 16:20:19.205925 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:19.205774    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" event={"ID":"9d136da5e3ede6720eaff51fc218626d","Type":"ContainerDied","Data":"631975076ad168af31b1902e088165d927950ca6b1ed97ebb4da7be8615722bd"}
Jul 28 16:20:19.214245 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:19.214212    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" event={"ID":"a87de93e34c8837496bc8bb7c2576ca1","Type":"ContainerStarted","Data":"7af70df5e9bc22450b7ca74ebd57eba98f9e8e0b66c14c0e572cfb569746736a"}
Jul 28 16:20:20.168327 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.168281    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:20.168847 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.168439    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:20.168908 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.168864    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:20.169018 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.168978    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:20.248658 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.248378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" event={"ID":"9d136da5e3ede6720eaff51fc218626d","Type":"ContainerStarted","Data":"16279f3f651438251886cfe186113c48060f9c329f04c881debc2cf78b28f99e"}
Jul 28 16:20:20.267003 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.266171    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-120.ec2.internal" podStartSLOduration=3.266156793 podStartE2EDuration="3.266156793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:20.265793329 +0000 UTC m=+5.659109871" watchObservedRunningTime="2026-07-28 16:20:20.266156793 +0000 UTC m=+5.659473334"
Jul 28 16:20:20.267003 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.266674    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-120.ec2.internal" podStartSLOduration=3.266647274 podStartE2EDuration="3.266647274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:19.240789384 +0000 UTC m=+4.634105919" watchObservedRunningTime="2026-07-28 16:20:20.266647274 +0000 UTC m=+5.659963819"
Jul 28 16:20:20.630600 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.629831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:20.630600 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.629898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:20.630600 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.630031    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:20.630600 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.630093    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:24.630074125 +0000 UTC m=+10.023390652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:20.630600 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.630156    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:20.630600 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.630190    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:20:24.630178693 +0000 UTC m=+10.023495215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:20.731019 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:20.730926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:20.731190 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.731127    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:20.731190 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.731154    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:20.731190 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.731169    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:20.731360 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:20.731236    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:24.73121692 +0000 UTC m=+10.124533453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:21.169275 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:21.169248    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:21.169758 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:21.169370    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:22.168318 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:22.168281    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:22.168528 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:22.168429    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:22.168887 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:22.168863    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:22.168982 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:22.168968    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:23.168808 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:23.168171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:23.168808 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:23.168311    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:24.167333 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:24.167299    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:24.167516 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.167448    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:24.167827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:24.167809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:24.167967 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.167917    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:24.660995 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:24.660870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:24.660995 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:24.660975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:24.661470 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.661065    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:24.661470 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.661112    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:20:32.661098081 +0000 UTC m=+18.054414600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:24.661575 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.661518    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:24.661614 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.661581    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:32.661563202 +0000 UTC m=+18.054879726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:24.762119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:24.762080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:24.762324 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.762299    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:24.762404 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.762327    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:24.762404 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.762344    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:24.762538 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:24.762404    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:32.762385121 +0000 UTC m=+18.155701640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:25.168901 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:25.168874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:25.169043 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:25.168958    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:26.167800 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:26.167763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:26.168302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:26.167810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:26.168302 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:26.167882    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:26.168302 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:26.167967    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:27.167979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:27.167508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:27.167979 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:27.167638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:28.168078 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:28.168039    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:28.168078 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:28.168064    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:28.168531 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:28.168162    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:28.168531 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:28.168278    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:29.167464 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:29.167436    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:29.167645 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:29.167555    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:30.167763 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:30.167727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:30.168236 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:30.167835    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:30.168236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:30.167896    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:30.168236 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:30.168006    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:31.168623 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:31.168588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:31.169029 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:31.168718    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:32.167879 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:32.167850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:32.168051 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:32.167929    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:32.168051 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.168005    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:32.168162 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.168142    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:32.729720 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:32.729683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:32.730146 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:32.729749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:32.730146 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.729846    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:32.730146 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.729851    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:32.730146 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.729895    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:48.729880773 +0000 UTC m=+34.123197290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:32.730146 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.729919    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:20:48.729900433 +0000 UTC m=+34.123216970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:32.830603 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:32.830575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:32.830754 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.830692    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:32.830754 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.830712    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:32.830754 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.830722    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:32.830912 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:32.830775    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:48.830759406 +0000 UTC m=+34.224075936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:33.167805 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:33.167726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:33.167959 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:33.167864    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:34.167964 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:34.167921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:34.168424 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:34.168051    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:34.168424 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:34.167939    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:34.168559 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:34.168477    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:35.169185 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:35.169150    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:35.169660 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:35.169258    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:36.167825 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:36.167798    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:36.168016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:36.167799    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:36.168016 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:36.167897    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:36.168016 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:36.167976    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:37.167989 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.167800    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:37.168323 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:37.168096    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:37.281987 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.281954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75vvp" event={"ID":"5785ce30-a7d4-4670-ba3a-fb7b162a1fec","Type":"ContainerStarted","Data":"9df6866a5fd40c29a597d736a9d833fc92d3a6d5814b507216363ae4a6e0120b"}
Jul 28 16:20:37.283490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.283464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwffh" event={"ID":"c1bca669-4acf-49c5-8686-5928990e6e9f","Type":"ContainerStarted","Data":"9905056cb3c7a667437e40edcd86ef8d41d07a873fd674ab63a0ecc0c60d3cb4"}
Jul 28 16:20:37.284842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.284818    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mf4gt" event={"ID":"34b01b86-43a7-4891-aa95-8bd3ce6a0bf6","Type":"ContainerStarted","Data":"51e0e3b0f6ff5592bc1b0d14c7c2a85e13351cf9ee85e57151d4a46131d46dcc"}
Jul 28 16:20:37.286450 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.286428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnvwk" event={"ID":"07640db5-ab1c-4797-870b-4b9f3767dbe1","Type":"ContainerStarted","Data":"2c2ea1d47d8c92123d28616c103886129bc5da83afe6bf6743e3838ed602c3a5"}
Jul 28 16:20:37.287848 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.287830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc" event={"ID":"9deb4699-aee3-4e43-a1fa-0560f6328774","Type":"ContainerStarted","Data":"af08ce5da3a159cc4ac32474387f99a0beb09fe09f1ed37178904f524f0a05e7"}
Jul 28 16:20:37.289291 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.289267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vm6qz" event={"ID":"b7a9e889-bb8a-4c9d-8fe0-12f2200c179c","Type":"ContainerStarted","Data":"3679a9f333ee562380aefac96cc777611ed577a3c6d5e32a406e22daa9330494"}
Jul 28 16:20:37.290976 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.290959    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:20:37.291356 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.291332    2573 generic.go:356] "Generic (PLEG): container finished" podID="0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97" containerID="c55234e8cce8609eec9aa9acdb087f3e896a666c6b4df1867551f68f580e33d9" exitCode=1
Jul 28 16:20:37.291439 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.291408    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerDied","Data":"c55234e8cce8609eec9aa9acdb087f3e896a666c6b4df1867551f68f580e33d9"}
Jul 28 16:20:37.291490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.291480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"a8bab558564ee92e565b5916dc24aa967702f9d7a4c1562b4a4f237bd520eb07"}
Jul 28 16:20:37.293078 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.293056    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9" containerID="5e5f13bc5a5b23240bcd01c58e1676224670124af023c0e8765b7cbf06e799d3" exitCode=0
Jul 28 16:20:37.293156 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.293088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerDied","Data":"5e5f13bc5a5b23240bcd01c58e1676224670124af023c0e8765b7cbf06e799d3"}
Jul 28 16:20:37.301063 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.301021    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-75vvp" podStartSLOduration=2.573423644 podStartE2EDuration="21.301008575s" totalImagesPullingTime="18.727584931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:37.300736857 +0000 UTC m=+22.694053396" watchObservedRunningTime="2026-07-28 16:20:37.301008575 +0000 UTC m=+22.694325168"
Jul 28 16:20:37.333455 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.333402    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wwffh" podStartSLOduration=2.555883872 podStartE2EDuration="21.333368579s" totalImagesPullingTime="18.777484707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:37.316194199 +0000 UTC m=+22.709510738" watchObservedRunningTime="2026-07-28 16:20:37.333368579 +0000 UTC m=+22.726685118"
Jul 28 16:20:37.362888 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.362848    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vm6qz" podStartSLOduration=2.5889664740000002 podStartE2EDuration="21.362833108s" totalImagesPullingTime="18.773866634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:37.333527812 +0000 UTC m=+22.726844333" watchObservedRunningTime="2026-07-28 16:20:37.362833108 +0000 UTC m=+22.756149701"
Jul 28 16:20:37.395536 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.395407    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mf4gt" podStartSLOduration=2.598317141 podStartE2EDuration="21.395391576s" totalImagesPullingTime="18.797074435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:37.380718912 +0000 UTC m=+22.774035451" watchObservedRunningTime="2026-07-28 16:20:37.395391576 +0000 UTC m=+22.788708106"
Jul 28 16:20:37.395938 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:37.395912    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wnvwk" podStartSLOduration=2.661543657 podStartE2EDuration="21.395904377s" totalImagesPullingTime="18.73436072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:37.395606892 +0000 UTC m=+22.788923409" watchObservedRunningTime="2026-07-28 16:20:37.395904377 +0000 UTC m=+22.789220916"
Jul 28 16:20:38.168215 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.168179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:38.168215 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.168223    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:38.169052 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:38.168359    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:38.169052 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:38.168471    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:38.298365 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.298342    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:20:38.298819 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.298791    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"898badc2a61dbbc90e2b4fdd6a4f7dda3389c6e7f8ec4ec69b02f04f6e6aec74"}
Jul 28 16:20:38.298922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.298852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"2d6c27e1b288b7ef65b397ef8a20e4cd9a0c41de3ca98dfbe754e706d4de0461"}
Jul 28 16:20:38.298922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.298892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"7025bb84709796eeb39b626eaa8de127352f47b56821261fd69d190d2ad5300b"}
Jul 28 16:20:38.298991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.298922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"1d82f3833b2a4741ac78455bfaedebf0d8e9713dd8f8a697846572110e6b3da3"}
Jul 28 16:20:38.634762 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:38.634742    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 16:20:39.049494 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.049366    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T16:20:38.634759272Z","UUID":"1bfafca2-2839-40bd-9979-99c67ac5a16b","Handler":null,"Name":"","Endpoint":""}
Jul 28 16:20:39.050936 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.050913    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 16:20:39.051067 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.050956    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 16:20:39.168144 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.168116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:39.168313 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:39.168229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:39.304943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.304864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc" event={"ID":"9deb4699-aee3-4e43-a1fa-0560f6328774","Type":"ContainerStarted","Data":"f08066e8ccd21e1fe0ffefa1b2416ac93f6f06fbc11dfc4d1e2bd95e5a366444"}
Jul 28 16:20:39.307445 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.307036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xcnfm" event={"ID":"19f4b4cc-e6e5-46ce-89a7-12af2d4f4e53","Type":"ContainerStarted","Data":"6c678a072af97ab3c5dd45c6a4a08fe4501a72d9e02171fd16de740c629b6dc6"}
Jul 28 16:20:39.323948 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:39.323907    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xcnfm" podStartSLOduration=4.542829355 podStartE2EDuration="23.323894651s" totalImagesPullingTime="18.781065296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:39.323522117 +0000 UTC m=+24.716838657" watchObservedRunningTime="2026-07-28 16:20:39.323894651 +0000 UTC m=+24.717211189"
Jul 28 16:20:40.167874 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:40.167844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:40.167874 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:40.167881    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:40.168088 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:40.167966    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:40.168164 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:40.168138    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:40.312069 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:40.312009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc" event={"ID":"9deb4699-aee3-4e43-a1fa-0560f6328774","Type":"ContainerStarted","Data":"f5284e68dfdc5100e5e9ddb7a0e4fb200406838e393a574ede5f377d25cc6fd9"}
Jul 28 16:20:40.331419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:40.331361    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4j9tc" podStartSLOduration=3.276971285 podStartE2EDuration="24.331342314s" totalImagesPullingTime="21.054371029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:40.331323022 +0000 UTC m=+25.724639565" watchObservedRunningTime="2026-07-28 16:20:40.331342314 +0000 UTC m=+25.724658853"
Jul 28 16:20:40.751403 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:40.751357    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:40.752557 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:40.752538    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:41.167362 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:41.167296    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:41.167520 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:41.167409    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:41.317280 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:41.317107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:20:41.317749 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:41.317720    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"614a64ae140ffc6f5534e28749570c7b8b093bf0acfab73a86e8fce5fbc17078"}
Jul 28 16:20:41.682525 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:41.682463    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:41.683005 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:41.682990    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnvwk"
Jul 28 16:20:42.167593 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:42.167517    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:42.167730 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:42.167602    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:42.167730 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:42.167664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:42.167839 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:42.167771    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:42.321622 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:42.321578    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9" containerID="438a660ea05b75a797361bbebc9629e14cc5253dea580fe836c02e099439b05a" exitCode=0
Jul 28 16:20:42.322194 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:42.321658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerDied","Data":"438a660ea05b75a797361bbebc9629e14cc5253dea580fe836c02e099439b05a"}
Jul 28 16:20:43.168196 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:43.168123    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:43.168338 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:43.168224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:44.167655 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.167472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:44.168192 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.167472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:44.168192 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:44.167723    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:44.168192 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:44.167782    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:44.329494 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.329467    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:20:44.329827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.329792    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"5604ee2be3a36434c00269157a3444e29f9e2dba9484714d79c1e9bc8e30fa66"}
Jul 28 16:20:44.330098 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.330076    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:44.330205 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.330150    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:44.330205 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.330192    2573 scope.go:117] "RemoveContainer" containerID="c55234e8cce8609eec9aa9acdb087f3e896a666c6b4df1867551f68f580e33d9"
Jul 28 16:20:44.330278 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.330199    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:44.332329 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.332306    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9" containerID="ecb1f8a49f55b817dbb581d75a32bff5a72c2c9ba36fad87c6790ad2c82dbe61" exitCode=0
Jul 28 16:20:44.332452 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.332341    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerDied","Data":"ecb1f8a49f55b817dbb581d75a32bff5a72c2c9ba36fad87c6790ad2c82dbe61"}
Jul 28 16:20:44.347706 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.347687    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:44.347822 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:44.347811    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:20:45.169873 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:45.169847    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:45.170263 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:45.169959    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:45.337516 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:45.337490    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:20:45.337873 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:45.337848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" event={"ID":"0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97","Type":"ContainerStarted","Data":"e01e3c227c360db0fa7458d8e9fed11aac2a3248c6888d300eb1467cb41a160c"}
Jul 28 16:20:45.340112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:45.340091    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9" containerID="b9d24e0fa78c411dfd38c5f5d549e2ffd8e3b7c99b5f31b46461e242d71bfbbb" exitCode=0
Jul 28 16:20:45.340220 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:45.340129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerDied","Data":"b9d24e0fa78c411dfd38c5f5d549e2ffd8e3b7c99b5f31b46461e242d71bfbbb"}
Jul 28 16:20:45.370444 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:45.370405    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4" podStartSLOduration=10.537339779 podStartE2EDuration="29.370391984s" totalImagesPullingTime="18.833052205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:45.36996176 +0000 UTC m=+30.763278301" watchObservedRunningTime="2026-07-28 16:20:45.370391984 +0000 UTC m=+30.763708514"
Jul 28 16:20:46.167532 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:46.167495    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:46.167715 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:46.167537    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:46.167715 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:46.167641    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:46.167715 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:46.167650    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:47.167990 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:47.167958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:47.168466 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:47.168083    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:48.167615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.167587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:48.167778 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.167691    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:48.167778 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.167737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:48.167878 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.167816    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:48.658752 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.658544    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zx225"]
Jul 28 16:20:48.659175 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.658828    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:48.659175 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.658934    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:48.671606 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.671574    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nfzgs"]
Jul 28 16:20:48.671757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.671705    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:48.671845 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.671822    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:48.672098 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.672079    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6h5n9"]
Jul 28 16:20:48.672167 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.672155    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:48.672249 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.672227    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:48.751081 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.751005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:48.751081 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.751064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:48.751301 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.751178    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:48.751301 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.751182    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:48.751301 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.751243    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret podName:cb496c43-a63c-4e6c-9cd7-1c74d7009d2c nodeName:}" failed. No retries permitted until 2026-07-28 16:21:20.751224856 +0000 UTC m=+66.144541374 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret") pod "global-pull-secret-syncer-zx225" (UID: "cb496c43-a63c-4e6c-9cd7-1c74d7009d2c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:20:48.751301 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.751260    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs podName:6ee98b90-1ba1-4013-bf30-8b89f1a028a6 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:20.751253008 +0000 UTC m=+66.144569526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs") pod "network-metrics-daemon-nfzgs" (UID: "6ee98b90-1ba1-4013-bf30-8b89f1a028a6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:20:48.851795 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:48.851759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:48.851973 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.851875    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:20:48.851973 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.851889    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:20:48.851973 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.851899    2573 projected.go:196] Error preparing data for projected volume kube-api-access-229rj for pod openshift-network-diagnostics/network-check-target-6h5n9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:48.851973 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:48.851946    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj podName:e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57 nodeName:}" failed. No retries permitted until 2026-07-28 16:21:20.851933965 +0000 UTC m=+66.245250483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-229rj" (UniqueName: "kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj") pod "network-check-target-6h5n9" (UID: "e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:20:50.167302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:50.167281    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:50.167644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:50.167309    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:50.167644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:50.167285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:50.167644 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:50.167398    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:50.167644 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:50.167465    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:50.167644 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:50.167558    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:50.353366 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:50.353307    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9" containerID="986459f62d75a641f6b85378915ba1d3d007cf960340737f84feff6510397751" exitCode=0
Jul 28 16:20:50.353498 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:50.353362    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerDied","Data":"986459f62d75a641f6b85378915ba1d3d007cf960340737f84feff6510397751"}
Jul 28 16:20:51.358125 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:51.358099    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9" containerID="7f33e3343de05e4f2b5356d0566fa03867c3781c861a31bf9865a20b3017b4e6" exitCode=0
Jul 28 16:20:51.358708 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:51.358132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerDied","Data":"7f33e3343de05e4f2b5356d0566fa03867c3781c861a31bf9865a20b3017b4e6"}
Jul 28 16:20:52.168057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.168024    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:52.168211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.168118    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:52.168211 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:52.168128    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6h5n9" podUID="e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57"
Jul 28 16:20:52.168211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.168140    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:52.168306 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:52.168217    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfzgs" podUID="6ee98b90-1ba1-4013-bf30-8b89f1a028a6"
Jul 28 16:20:52.168338 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:20:52.168317    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zx225" podUID="cb496c43-a63c-4e6c-9cd7-1c74d7009d2c"
Jul 28 16:20:52.363125 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.363092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" event={"ID":"ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9","Type":"ContainerStarted","Data":"0bf5d042e9d0f4420b81fe91d62160ba5627669e57f21cd4f2d06069f7579400"}
Jul 28 16:20:52.388350 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.388285    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6hfsv" podStartSLOduration=6.8584604890000005 podStartE2EDuration="36.388266833s" totalImagesPullingTime="29.529806344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:52.386317917 +0000 UTC m=+37.779634457" watchObservedRunningTime="2026-07-28 16:20:52.388266833 +0000 UTC m=+37.781583379"
Jul 28 16:20:52.910643 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.910578    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-120.ec2.internal" event="NodeReady"
Jul 28 16:20:52.910773 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.910706    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 16:20:52.962763 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.962730    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qzdmk"]
Jul 28 16:20:52.976856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.976836    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:52.976856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.976845    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bfj8q"]
Jul 28 16:20:52.979684 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.979664    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 16:20:52.979850 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.979832    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 16:20:52.979914 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.979899    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qctb2\""
Jul 28 16:20:52.980437 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.980419    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 16:20:52.989405 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.989385    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qzdmk"]
Jul 28 16:20:52.989480 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.989415    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bfj8q"]
Jul 28 16:20:52.989480 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.989436    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4lp5c"]
Jul 28 16:20:52.989578 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.989479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:52.992185 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.992164    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 16:20:52.992280 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.992218    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 16:20:52.992280 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.992237    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 16:20:52.992280 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.992240    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s22cc\""
Jul 28 16:20:52.992406 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:52.992317    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 16:20:53.001641 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.001580    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4lp5c"]
Jul 28 16:20:53.001731 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.001641    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.003590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.003564    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 16:20:53.003590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.003578    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 16:20:53.003719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.003646    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v7jht\""
Jul 28 16:20:53.083619 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f-cert\") pod \"ingress-canary-qzdmk\" (UID: \"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f\") " pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.083619 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56bg\" (UniqueName: \"kubernetes.io/projected/6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f-kube-api-access-j56bg\") pod \"ingress-canary-qzdmk\" (UID: \"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f\") " pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.083841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7599f6b8-1ce8-4340-973e-367c1b8c778d-data-volume\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.083841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083654    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7599f6b8-1ce8-4340-973e-367c1b8c778d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.083841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7599f6b8-1ce8-4340-973e-367c1b8c778d-crio-socket\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.083841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7599f6b8-1ce8-4340-973e-367c1b8c778d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.083841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.083826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cd57\" (UniqueName: \"kubernetes.io/projected/7599f6b8-1ce8-4340-973e-367c1b8c778d-kube-api-access-8cd57\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184131 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7599f6b8-1ce8-4340-973e-367c1b8c778d-crio-socket\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184131 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-config-volume\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.184131 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-metrics-tls\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.184335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184146    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgrz\" (UniqueName: \"kubernetes.io/projected/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-kube-api-access-pcgrz\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.184335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7599f6b8-1ce8-4340-973e-367c1b8c778d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cd57\" (UniqueName: \"kubernetes.io/projected/7599f6b8-1ce8-4340-973e-367c1b8c778d-kube-api-access-8cd57\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-tmp-dir\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.184335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184313    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f-cert\") pod \"ingress-canary-qzdmk\" (UID: \"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f\") " pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.184580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56bg\" (UniqueName: \"kubernetes.io/projected/6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f-kube-api-access-j56bg\") pod \"ingress-canary-qzdmk\" (UID: \"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f\") " pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.184580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184352    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7599f6b8-1ce8-4340-973e-367c1b8c778d-crio-socket\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7599f6b8-1ce8-4340-973e-367c1b8c778d-data-volume\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7599f6b8-1ce8-4340-973e-367c1b8c778d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184758 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7599f6b8-1ce8-4340-973e-367c1b8c778d-data-volume\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.184930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.184914    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7599f6b8-1ce8-4340-973e-367c1b8c778d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.188263 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.188237    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7599f6b8-1ce8-4340-973e-367c1b8c778d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.188468 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.188452    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f-cert\") pod \"ingress-canary-qzdmk\" (UID: \"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f\") " pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.194685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.194663    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56bg\" (UniqueName: \"kubernetes.io/projected/6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f-kube-api-access-j56bg\") pod \"ingress-canary-qzdmk\" (UID: \"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f\") " pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.194751 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.194686    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cd57\" (UniqueName: \"kubernetes.io/projected/7599f6b8-1ce8-4340-973e-367c1b8c778d-kube-api-access-8cd57\") pod \"insights-runtime-extractor-bfj8q\" (UID: \"7599f6b8-1ce8-4340-973e-367c1b8c778d\") " pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.285044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-config-volume\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.285044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285047    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-metrics-tls\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.285300 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgrz\" (UniqueName: \"kubernetes.io/projected/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-kube-api-access-pcgrz\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.285300 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-tmp-dir\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.285494 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285475    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-tmp-dir\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.285611 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285593    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-config-volume\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.285677 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.285660    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qzdmk"
Jul 28 16:20:53.287296 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.287276    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-metrics-tls\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.295227 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.295207    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgrz\" (UniqueName: \"kubernetes.io/projected/ad54d500-f594-4094-8d9d-09e7c4a0f8fd-kube-api-access-pcgrz\") pod \"dns-default-4lp5c\" (UID: \"ad54d500-f594-4094-8d9d-09e7c4a0f8fd\") " pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.297159 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.297139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bfj8q"
Jul 28 16:20:53.308970 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.308941    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:20:53.473587 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.473318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qzdmk"]
Jul 28 16:20:53.477553 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:53.477519    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6b08cf_1d43_4000_9fb3_592b2bbd5e6f.slice/crio-8be1e2092219b89d4eabe5d570ba0f5dc3beb0894697a13baf9e109986a879fb WatchSource:0}: Error finding container 8be1e2092219b89d4eabe5d570ba0f5dc3beb0894697a13baf9e109986a879fb: Status 404 returned error can't find the container with id 8be1e2092219b89d4eabe5d570ba0f5dc3beb0894697a13baf9e109986a879fb
Jul 28 16:20:53.478951 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.478722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bfj8q"]
Jul 28 16:20:53.482994 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:53.482973    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7599f6b8_1ce8_4340_973e_367c1b8c778d.slice/crio-e622be9a8cba52eb4329e7cfe8b65a0927730422def7eb60517fcf2173d214cf WatchSource:0}: Error finding container e622be9a8cba52eb4329e7cfe8b65a0927730422def7eb60517fcf2173d214cf: Status 404 returned error can't find the container with id e622be9a8cba52eb4329e7cfe8b65a0927730422def7eb60517fcf2173d214cf
Jul 28 16:20:53.488244 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:53.488219    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4lp5c"]
Jul 28 16:20:53.494172 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:20:53.494151    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad54d500_f594_4094_8d9d_09e7c4a0f8fd.slice/crio-fce2392cb55ec0f6cba312bd0be48f6497f6e986c277ca9a1e43c32f92e64ff8 WatchSource:0}: Error finding container fce2392cb55ec0f6cba312bd0be48f6497f6e986c277ca9a1e43c32f92e64ff8: Status 404 returned error can't find the container with id fce2392cb55ec0f6cba312bd0be48f6497f6e986c277ca9a1e43c32f92e64ff8
Jul 28 16:20:54.168132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.168094    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:20:54.168606 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.168561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:20:54.168956 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.168940    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:20:54.171499 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.171448    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:20:54.171606 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.171554    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p4vdz\""
Jul 28 16:20:54.171721 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.171700    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:20:54.171871 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.171854    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-njslf\""
Jul 28 16:20:54.171937 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.171871    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:20:54.171991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.171979    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:20:54.369732 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.369692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qzdmk" event={"ID":"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f","Type":"ContainerStarted","Data":"8be1e2092219b89d4eabe5d570ba0f5dc3beb0894697a13baf9e109986a879fb"}
Jul 28 16:20:54.372613 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.372548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lp5c" event={"ID":"ad54d500-f594-4094-8d9d-09e7c4a0f8fd","Type":"ContainerStarted","Data":"fce2392cb55ec0f6cba312bd0be48f6497f6e986c277ca9a1e43c32f92e64ff8"}
Jul 28 16:20:54.375385 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:20:54.375305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfj8q" event={"ID":"7599f6b8-1ce8-4340-973e-367c1b8c778d","Type":"ContainerStarted","Data":"e622be9a8cba52eb4329e7cfe8b65a0927730422def7eb60517fcf2173d214cf"}
Jul 28 16:21:00.391641 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:00.391612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qzdmk" event={"ID":"6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f","Type":"ContainerStarted","Data":"6d2596fd6f3999ffbf41ca6590ef98530202f5ee510ba2f47a4c7403e7935d08"}
Jul 28 16:21:00.393079 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:00.393054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lp5c" event={"ID":"ad54d500-f594-4094-8d9d-09e7c4a0f8fd","Type":"ContainerStarted","Data":"3a637bdd2ac96de52af4342ef9c647524637118150be893e1dc5d560feb42551"}
Jul 28 16:21:00.393162 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:00.393095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4lp5c" event={"ID":"ad54d500-f594-4094-8d9d-09e7c4a0f8fd","Type":"ContainerStarted","Data":"d70e97c19c208c5e01feac3cfa4810701ed1aad04c4d512cd78f886566341657"}
Jul 28 16:21:00.411236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:00.411179    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:21:00.430225 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:00.430187    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qzdmk" podStartSLOduration=1.883285308 podStartE2EDuration="8.430175012s" totalImagesPullingTime="6.546889704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:00.429083048 +0000 UTC m=+45.822399586" watchObservedRunningTime="2026-07-28 16:21:00.430175012 +0000 UTC m=+45.823491550"
Jul 28 16:21:00.447920 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:00.447762    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4lp5c" podStartSLOduration=1.916874145 podStartE2EDuration="8.44774754s" totalImagesPullingTime="6.530873395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:00.447644975 +0000 UTC m=+45.840961514" watchObservedRunningTime="2026-07-28 16:21:00.44774754 +0000 UTC m=+45.841064074"
Jul 28 16:21:01.397963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:01.397920    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfj8q" event={"ID":"7599f6b8-1ce8-4340-973e-367c1b8c778d","Type":"ContainerStarted","Data":"6f9379a70d4a6e5f249b19bd8d53958bf48f3344b4399462d4d664598aa636d6"}
Jul 28 16:21:05.413776 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:05.413739    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfj8q" event={"ID":"7599f6b8-1ce8-4340-973e-367c1b8c778d","Type":"ContainerStarted","Data":"13fa75b868b18e80b6df9c7f381299b0147dc75771eb4340e6640f57c78e2aca"}
Jul 28 16:21:06.149997 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.149961    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-9cczv"]
Jul 28 16:21:06.173165 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.173138    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-9cczv"]
Jul 28 16:21:06.173320 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.173226    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.175760 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.175736    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 16:21:06.176279 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.176259    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 16:21:06.176279 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.176257    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b5swr\""
Jul 28 16:21:06.176473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.176335    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 16:21:06.176473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.176263    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 16:21:06.176798 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.176782    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 16:21:06.177226 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.177208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.177308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.177259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.177403 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.177320    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.177461 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.177400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffh9\" (UniqueName: \"kubernetes.io/projected/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-kube-api-access-9ffh9\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.278277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.278249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.278446 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.278314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.278446 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.278347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.278446 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.278413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffh9\" (UniqueName: \"kubernetes.io/projected/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-kube-api-access-9ffh9\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.279101 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.279079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.282271 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.282249    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.282353 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.282326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.287035 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.287013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffh9\" (UniqueName: \"kubernetes.io/projected/8f0cac8e-0b86-4709-b5af-d43d1b8813fa-kube-api-access-9ffh9\") pod \"prometheus-operator-84d69f794d-9cczv\" (UID: \"8f0cac8e-0b86-4709-b5af-d43d1b8813fa\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.484364 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.484325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv"
Jul 28 16:21:06.661881 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:06.661851    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-9cczv"]
Jul 28 16:21:06.665326 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:06.665289    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0cac8e_0b86_4709_b5af_d43d1b8813fa.slice/crio-6527ea38e20695fb0cffe56b2bcfc3ed290bf7a50011a38644a885bf64eba0a2 WatchSource:0}: Error finding container 6527ea38e20695fb0cffe56b2bcfc3ed290bf7a50011a38644a885bf64eba0a2: Status 404 returned error can't find the container with id 6527ea38e20695fb0cffe56b2bcfc3ed290bf7a50011a38644a885bf64eba0a2
Jul 28 16:21:07.422086 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:07.422055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv" event={"ID":"8f0cac8e-0b86-4709-b5af-d43d1b8813fa","Type":"ContainerStarted","Data":"6527ea38e20695fb0cffe56b2bcfc3ed290bf7a50011a38644a885bf64eba0a2"}
Jul 28 16:21:09.430785 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:09.430749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bfj8q" event={"ID":"7599f6b8-1ce8-4340-973e-367c1b8c778d","Type":"ContainerStarted","Data":"9e0e686b137178c4dc40856e50e454406fff6e00154c06f43b5bdb92163cf5e9"}
Jul 28 16:21:10.400112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.400076    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4lp5c"
Jul 28 16:21:10.429301 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.429249    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bfj8q" podStartSLOduration=3.361609176 podStartE2EDuration="18.429231184s" totalImagesPullingTime="15.067622008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:09.459786696 +0000 UTC m=+54.853103234" watchObservedRunningTime="2026-07-28 16:21:10.429231184 +0000 UTC m=+55.822547724"
Jul 28 16:21:10.435503 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.435474    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv" event={"ID":"8f0cac8e-0b86-4709-b5af-d43d1b8813fa","Type":"ContainerStarted","Data":"ae8b637caa3ae07f72aa6666fd66bef999b719f632b185578eaeb45eb72c983f"}
Jul 28 16:21:10.435900 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.435523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv" event={"ID":"8f0cac8e-0b86-4709-b5af-d43d1b8813fa","Type":"ContainerStarted","Data":"dc377bfe39c15df83b2395305a5351e48b90aea20d4778f46a301c4e5e959966"}
Jul 28 16:21:10.892669 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.892610    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-9cczv" podStartSLOduration=1.934455252 podStartE2EDuration="4.892585155s" totalImagesPullingTime="2.958129903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:10.471393511 +0000 UTC m=+55.864710048" watchObservedRunningTime="2026-07-28 16:21:10.892585155 +0000 UTC m=+56.285901695"
Jul 28 16:21:10.893334 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.893315    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-l228b"]
Jul 28 16:21:10.895668 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.895654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:10.897906 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.897887    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nw4vm\""
Jul 28 16:21:10.898478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.898463    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 16:21:10.898583 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.898567    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 16:21:10.908563 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.908539    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-l228b"]
Jul 28 16:21:10.912658 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:10.912638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jh9k\" (UniqueName: \"kubernetes.io/projected/68db1376-c928-43d3-af39-84d019ff6212-kube-api-access-9jh9k\") pod \"downloads-7c97dc4555-l228b\" (UID: \"68db1376-c928-43d3-af39-84d019ff6212\") " pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:11.013306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:11.013270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jh9k\" (UniqueName: \"kubernetes.io/projected/68db1376-c928-43d3-af39-84d019ff6212-kube-api-access-9jh9k\") pod \"downloads-7c97dc4555-l228b\" (UID: \"68db1376-c928-43d3-af39-84d019ff6212\") " pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:11.026004 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:11.025975    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jh9k\" (UniqueName: \"kubernetes.io/projected/68db1376-c928-43d3-af39-84d019ff6212-kube-api-access-9jh9k\") pod \"downloads-7c97dc4555-l228b\" (UID: \"68db1376-c928-43d3-af39-84d019ff6212\") " pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:11.204507 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:11.204416    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:11.331225 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:11.331191    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-l228b"]
Jul 28 16:21:11.334218 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:11.334185    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68db1376_c928_43d3_af39_84d019ff6212.slice/crio-42fc7518333cfc26a644112ea648d8a8d3d853b839fea22ad17a903efbf67719 WatchSource:0}: Error finding container 42fc7518333cfc26a644112ea648d8a8d3d853b839fea22ad17a903efbf67719: Status 404 returned error can't find the container with id 42fc7518333cfc26a644112ea648d8a8d3d853b839fea22ad17a903efbf67719
Jul 28 16:21:11.439867 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:11.439825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-l228b" event={"ID":"68db1376-c928-43d3-af39-84d019ff6212","Type":"ContainerStarted","Data":"42fc7518333cfc26a644112ea648d8a8d3d853b839fea22ad17a903efbf67719"}
Jul 28 16:21:12.753387 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.753334    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-njfjh"]
Jul 28 16:21:12.755669 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.755647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.761138 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.761108    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 16:21:12.761254 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.761188    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 16:21:12.761310 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.761188    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-drlz2\""
Jul 28 16:21:12.762126 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.762104    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 16:21:12.826020 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.825971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826206 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826206 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-sys\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826206 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-tls\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826206 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826158    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-textfile\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826206 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-root\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826206 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-metrics-client-ca\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826554 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-wtmp\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.826554 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.826277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db44p\" (UniqueName: \"kubernetes.io/projected/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-kube-api-access-db44p\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-tls\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-textfile\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-root\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-metrics-client-ca\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-wtmp\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db44p\" (UniqueName: \"kubernetes.io/projected/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-kube-api-access-db44p\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-root\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-sys\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927813 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-wtmp\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927813 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927552    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-sys\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927813 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-textfile\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.927979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.927962    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-metrics-client-ca\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.928243 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.928212    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.930124 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.930099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.930292 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.930265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-node-exporter-tls\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:12.938597 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:12.938574    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db44p\" (UniqueName: \"kubernetes.io/projected/bf71d79a-7ffe-470d-b6ee-45b2c1e221cf-kube-api-access-db44p\") pod \"node-exporter-njfjh\" (UID: \"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf\") " pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:13.067922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:13.067831    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njfjh"
Jul 28 16:21:13.079102 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:13.079065    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf71d79a_7ffe_470d_b6ee_45b2c1e221cf.slice/crio-238c4c8c03b9a6c8f01c7172fb9f56f2491737dd18a0a57aa1cd1891b5f65b3d WatchSource:0}: Error finding container 238c4c8c03b9a6c8f01c7172fb9f56f2491737dd18a0a57aa1cd1891b5f65b3d: Status 404 returned error can't find the container with id 238c4c8c03b9a6c8f01c7172fb9f56f2491737dd18a0a57aa1cd1891b5f65b3d
Jul 28 16:21:13.452047 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:13.451963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njfjh" event={"ID":"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf","Type":"ContainerStarted","Data":"238c4c8c03b9a6c8f01c7172fb9f56f2491737dd18a0a57aa1cd1891b5f65b3d"}
Jul 28 16:21:14.457057 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:14.457022    2573 generic.go:356] "Generic (PLEG): container finished" podID="bf71d79a-7ffe-470d-b6ee-45b2c1e221cf" containerID="32f5796ee7184a62f3a0d58012896914c56ff2844201f4cee88aed6b9e5f8dd9" exitCode=0
Jul 28 16:21:14.457453 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:14.457081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njfjh" event={"ID":"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf","Type":"ContainerDied","Data":"32f5796ee7184a62f3a0d58012896914c56ff2844201f4cee88aed6b9e5f8dd9"}
Jul 28 16:21:15.463088 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.463041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njfjh" event={"ID":"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf","Type":"ContainerStarted","Data":"f6268411a18685a4985701725d1b2426ed397ba406853ea51483357300cbd07b"}
Jul 28 16:21:15.463676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.463117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njfjh" event={"ID":"bf71d79a-7ffe-470d-b6ee-45b2c1e221cf","Type":"ContainerStarted","Data":"c26e84e457ecab56f819f7b32f269043b919b7aae82c9ed948ae742a7b1c5bf0"}
Jul 28 16:21:15.491282 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.491227    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-njfjh" podStartSLOduration=2.232448394 podStartE2EDuration="3.491213152s" totalImagesPullingTime="1.177145407s" totalInitContainerRuntime="81.619351ms" isStatefulPod=true podCreationTimestamp="2026-07-28 16:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:15.490247574 +0000 UTC m=+60.883564124" watchObservedRunningTime="2026-07-28 16:21:15.491213152 +0000 UTC m=+60.884529689"
Jul 28 16:21:15.742632 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.742596    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-555b4fff79-gxwmp"]
Jul 28 16:21:15.745922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.745897    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.748730 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.748704    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 16:21:15.748864 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.748712    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 16:21:15.748928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.748866    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9kztk\""
Jul 28 16:21:15.748968 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.748945    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 16:21:15.749081 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.749058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 16:21:15.749203 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.749188    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 16:21:15.749300 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.749283    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9lpuj1ign2l2i\""
Jul 28 16:21:15.761642 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.761613    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-555b4fff79-gxwmp"]
Jul 28 16:21:15.849603 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmv6\" (UniqueName: \"kubernetes.io/projected/da067996-34f2-4510-9dd1-4a5879f585ef-kube-api-access-8qmv6\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.849603 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.849856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.849856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-grpc-tls\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.849856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849795    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.849856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849814    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da067996-34f2-4510-9dd1-4a5879f585ef-metrics-client-ca\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.850027 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849904    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.850027 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.849938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-tls\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950531 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-grpc-tls\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950531 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950531 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da067996-34f2-4510-9dd1-4a5879f585ef-metrics-client-ca\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-tls\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950623    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmv6\" (UniqueName: \"kubernetes.io/projected/da067996-34f2-4510-9dd1-4a5879f585ef-kube-api-access-8qmv6\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.950694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.950650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.952103 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.952054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da067996-34f2-4510-9dd1-4a5879f585ef-metrics-client-ca\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.954030 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.953679    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-grpc-tls\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.954030 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.953935    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-tls\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.954030 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.953998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.954645 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.954626    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.955259 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.955237    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.955602 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.955582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/da067996-34f2-4510-9dd1-4a5879f585ef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:15.960956 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:15.960925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmv6\" (UniqueName: \"kubernetes.io/projected/da067996-34f2-4510-9dd1-4a5879f585ef-kube-api-access-8qmv6\") pod \"thanos-querier-555b4fff79-gxwmp\" (UID: \"da067996-34f2-4510-9dd1-4a5879f585ef\") " pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:16.057045 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:16.056969    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:16.214901 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:16.214872    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-555b4fff79-gxwmp"]
Jul 28 16:21:16.218546 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:16.218506    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda067996_34f2_4510_9dd1_4a5879f585ef.slice/crio-b01a3558a21560dd5528051b446aef77641d8b36fabaf142961a1fc4eb0951bd WatchSource:0}: Error finding container b01a3558a21560dd5528051b446aef77641d8b36fabaf142961a1fc4eb0951bd: Status 404 returned error can't find the container with id b01a3558a21560dd5528051b446aef77641d8b36fabaf142961a1fc4eb0951bd
Jul 28 16:21:16.354438 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:16.354338    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pszj4"
Jul 28 16:21:16.469111 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:16.469077    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"b01a3558a21560dd5528051b446aef77641d8b36fabaf142961a1fc4eb0951bd"}
Jul 28 16:21:17.407598 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.407534    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"]
Jul 28 16:21:17.409594 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.409566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:17.415165 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.415138    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zxn7m\""
Jul 28 16:21:17.415297 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.415219    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 16:21:17.429086 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.429057    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"]
Jul 28 16:21:17.464630 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.464592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xsbm\" (UniqueName: \"kubernetes.io/projected/09bd4ee5-6f10-426f-b022-24ae529e264f-kube-api-access-7xsbm\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:17.464819 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.464676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09bd4ee5-6f10-426f-b022-24ae529e264f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:17.566424 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.565651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xsbm\" (UniqueName: \"kubernetes.io/projected/09bd4ee5-6f10-426f-b022-24ae529e264f-kube-api-access-7xsbm\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:17.566424 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.565768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09bd4ee5-6f10-426f-b022-24ae529e264f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:17.566424 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:21:17.565898    2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 16:21:17.566424 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:21:17.565959    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09bd4ee5-6f10-426f-b022-24ae529e264f-monitoring-plugin-cert podName:09bd4ee5-6f10-426f-b022-24ae529e264f nodeName:}" failed. No retries permitted until 2026-07-28 16:21:18.065940203 +0000 UTC m=+63.459256738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/09bd4ee5-6f10-426f-b022-24ae529e264f-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-2hsb5" (UID: "09bd4ee5-6f10-426f-b022-24ae529e264f") : secret "monitoring-plugin-cert" not found
Jul 28 16:21:17.578193 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:17.578057    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xsbm\" (UniqueName: \"kubernetes.io/projected/09bd4ee5-6f10-426f-b022-24ae529e264f-kube-api-access-7xsbm\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:18.070231 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.070191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09bd4ee5-6f10-426f-b022-24ae529e264f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:18.078692 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.078658    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09bd4ee5-6f10-426f-b022-24ae529e264f-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-2hsb5\" (UID: \"09bd4ee5-6f10-426f-b022-24ae529e264f\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:18.322221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.322147    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:18.522153 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.522126    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"]
Jul 28 16:21:18.525517 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:18.525495    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bd4ee5_6f10_426f_b022_24ae529e264f.slice/crio-75ffa50bfe853e457772f4a2da29dd516f26e1bc1663e2460f3ad88effa398b2 WatchSource:0}: Error finding container 75ffa50bfe853e457772f4a2da29dd516f26e1bc1663e2460f3ad88effa398b2: Status 404 returned error can't find the container with id 75ffa50bfe853e457772f4a2da29dd516f26e1bc1663e2460f3ad88effa398b2
Jul 28 16:21:18.888054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.887973    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:21:18.891284 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.891257    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.895704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.895675    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mfc2l\""
Jul 28 16:21:18.895835 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.895727    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 16:21:18.895835 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.895739    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 16:21:18.895835 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.895691    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 16:21:18.895983 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.895676    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 16:21:18.896216 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896197    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 16:21:18.896216 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896211    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8bat0rlh9dpan\""
Jul 28 16:21:18.896414 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896396    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 16:21:18.896522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896502    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 16:21:18.896581 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896566    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 16:21:18.896639 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896590    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 16:21:18.896753 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.896736    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 16:21:18.897308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.897021    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 16:21:18.897308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.897112    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 16:21:18.897308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.897024    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 16:21:18.899962 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.899887    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 16:21:18.904215 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.904196    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 16:21:18.911676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.911656    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:21:18.980318 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980318 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980345    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2w5d\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-kube-api-access-g2w5d\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980512    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980813    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.980839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.981193 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.981193 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:18.981193 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:18.980919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w5d\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-kube-api-access-g2w5d\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081929 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081929 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081929 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.081929 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.081984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082539 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082147    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082539 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082539 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082539 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082346    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082539 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082406    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.082766 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.082584    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.083136 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.083112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.083586 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.083568    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.084861 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.084687    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.086349 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.085892    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.086349 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.085998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.086349 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.086217    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.086349 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.086312    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.086349 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.086348    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.088519 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.088498    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.088640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.088557    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.088772 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.088747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.088825 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.088790    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.088978 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.088940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.091128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.090854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.091128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.090922    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.092663 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.092639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2w5d\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-kube-api-access-g2w5d\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.093240 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.093220    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.204838 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.204345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:21:19.410068 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.409955    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:21:19.489821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.489617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"018fa2474ed19af5506bcc418e4713aedbfc31abb474a9bb04131ef026a9a019"}
Jul 28 16:21:19.489821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.489705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"94153173e9a474f39a4ac8a8f55f943d15b27912067d481d05810ad4e1ea238e"}
Jul 28 16:21:19.489821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.489757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"199ac434751cd84966ae5f7503343259b739dd195993a34ba44daf394f11317e"}
Jul 28 16:21:19.491341 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:19.491288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5" event={"ID":"09bd4ee5-6f10-426f-b022-24ae529e264f","Type":"ContainerStarted","Data":"75ffa50bfe853e457772f4a2da29dd516f26e1bc1663e2460f3ad88effa398b2"}
Jul 28 16:21:19.570899 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:19.570856    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec75ccde_c993_4ea8_8b4f_89e3159731f6.slice/crio-3015ab3edd15970aeb565c5c7bb179d74bb32b35051314431430b28e2410da27 WatchSource:0}: Error finding container 3015ab3edd15970aeb565c5c7bb179d74bb32b35051314431430b28e2410da27: Status 404 returned error can't find the container with id 3015ab3edd15970aeb565c5c7bb179d74bb32b35051314431430b28e2410da27
Jul 28 16:21:20.496265 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.496229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"3015ab3edd15970aeb565c5c7bb179d74bb32b35051314431430b28e2410da27"}
Jul 28 16:21:20.499679 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.499653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"7f51052ccc09c7db81da325bad45450ecf4f802715a741c200577f88ce01f950"}
Jul 28 16:21:20.499823 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.499711    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"fda2bfa3ceeca81d024d7a26674df50af23af147d369d83ebebf8276f81c1890"}
Jul 28 16:21:20.499823 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.499743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" event={"ID":"da067996-34f2-4510-9dd1-4a5879f585ef","Type":"ContainerStarted","Data":"fa98122423fa5e5aac12b63fb2273b6425eac140fdc7b63e1fb170275c98c5ff"}
Jul 28 16:21:20.499900 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.499853    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:20.524467 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.524357    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp" podStartSLOduration=2.320892739 podStartE2EDuration="5.524338142s" totalImagesPullingTime="3.203445403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:20.522788844 +0000 UTC m=+65.916105394" watchObservedRunningTime="2026-07-28 16:21:20.524338142 +0000 UTC m=+65.917654685"
Jul 28 16:21:20.797964 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.797886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:21:20.797964 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.797952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:21:20.800241 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.800215    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:21:20.800350 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.800216    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:21:20.812002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.811977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ee98b90-1ba1-4013-bf30-8b89f1a028a6-metrics-certs\") pod \"network-metrics-daemon-nfzgs\" (UID: \"6ee98b90-1ba1-4013-bf30-8b89f1a028a6\") " pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:21:20.812899 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.812854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb496c43-a63c-4e6c-9cd7-1c74d7009d2c-original-pull-secret\") pod \"global-pull-secret-syncer-zx225\" (UID: \"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c\") " pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:21:20.898737 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.898709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:21:20.900910 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.900890    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:21:20.908390 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.908354    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-njslf\""
Jul 28 16:21:20.911082 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.911064    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:21:20.913738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.913711    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zx225"
Jul 28 16:21:20.916836 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.916816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfzgs"
Jul 28 16:21:20.922688 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:20.922668    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-229rj\" (UniqueName: \"kubernetes.io/projected/e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57-kube-api-access-229rj\") pod \"network-check-target-6h5n9\" (UID: \"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57\") " pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:21:21.081506 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.081474    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zx225"]
Jul 28 16:21:21.184777 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.184742    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p4vdz\""
Jul 28 16:21:21.192858 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.192831    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:21:21.299884 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:21.299854    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb496c43_a63c_4e6c_9cd7_1c74d7009d2c.slice/crio-980b0411f079a33e74e07d0d8eb108fd7bc87fd2ce7338dc21f6048448688765 WatchSource:0}: Error finding container 980b0411f079a33e74e07d0d8eb108fd7bc87fd2ce7338dc21f6048448688765: Status 404 returned error can't find the container with id 980b0411f079a33e74e07d0d8eb108fd7bc87fd2ce7338dc21f6048448688765
Jul 28 16:21:21.456257 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.456126    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nfzgs"]
Jul 28 16:21:21.478586 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.478560    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6h5n9"]
Jul 28 16:21:21.491470 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:21.491441    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee98b90_1ba1_4013_bf30_8b89f1a028a6.slice/crio-dcc905703372c962164eb2358b3fd10e24606676b211fb486deed2cbeef3608f WatchSource:0}: Error finding container dcc905703372c962164eb2358b3fd10e24606676b211fb486deed2cbeef3608f: Status 404 returned error can't find the container with id dcc905703372c962164eb2358b3fd10e24606676b211fb486deed2cbeef3608f
Jul 28 16:21:21.503230 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:21:21.503205    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e6c0eb_781d_4bf5_92e0_6a4de60d9b57.slice/crio-306b3728428abe6d1d4dd0ff17ebbe9641f871c6c5a51396e070db693dfbd670 WatchSource:0}: Error finding container 306b3728428abe6d1d4dd0ff17ebbe9641f871c6c5a51396e070db693dfbd670: Status 404 returned error can't find the container with id 306b3728428abe6d1d4dd0ff17ebbe9641f871c6c5a51396e070db693dfbd670
Jul 28 16:21:21.505464 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.505014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5" event={"ID":"09bd4ee5-6f10-426f-b022-24ae529e264f","Type":"ContainerStarted","Data":"a584275375542d0cef3a623575554b9b25ae8e57e4c39904a9475a3bb7d062f6"}
Jul 28 16:21:21.505464 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.505251    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:21.506897 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.506869    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zx225" event={"ID":"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c","Type":"ContainerStarted","Data":"980b0411f079a33e74e07d0d8eb108fd7bc87fd2ce7338dc21f6048448688765"}
Jul 28 16:21:21.508284 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.508261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfzgs" event={"ID":"6ee98b90-1ba1-4013-bf30-8b89f1a028a6","Type":"ContainerStarted","Data":"dcc905703372c962164eb2358b3fd10e24606676b211fb486deed2cbeef3608f"}
Jul 28 16:21:21.510189 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.510167    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093" exitCode=0
Jul 28 16:21:21.510276 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.510253    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"}
Jul 28 16:21:21.511191 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.511114    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5"
Jul 28 16:21:21.528645 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:21.528597    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-2hsb5" podStartSLOduration=2.400808318 podStartE2EDuration="4.528577522s" totalImagesPullingTime="2.127769204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:21.526016257 +0000 UTC m=+66.919332795" watchObservedRunningTime="2026-07-28 16:21:21.528577522 +0000 UTC m=+66.921894062"
Jul 28 16:21:22.516617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:22.516574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6h5n9" event={"ID":"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57","Type":"ContainerStarted","Data":"306b3728428abe6d1d4dd0ff17ebbe9641f871c6c5a51396e070db693dfbd670"}
Jul 28 16:21:26.517975 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:26.517946    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-555b4fff79-gxwmp"
Jul 28 16:21:34.572438 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.570845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-l228b" event={"ID":"68db1376-c928-43d3-af39-84d019ff6212","Type":"ContainerStarted","Data":"b01714b368451148b304bc730114a21094964060162fd354f302f347ad7361af"}
Jul 28 16:21:34.572438 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.571517    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:34.575291 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.575260    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"}
Jul 28 16:21:34.575430 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.575336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"}
Jul 28 16:21:34.575430 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.575397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"}
Jul 28 16:21:34.577075 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.577051    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zx225" event={"ID":"cb496c43-a63c-4e6c-9cd7-1c74d7009d2c","Type":"ContainerStarted","Data":"da4edb74ebda9bd2bb69a7d41179ca7547af7d357545e251ed2fabb28f156513"}
Jul 28 16:21:34.579399 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.579325    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfzgs" event={"ID":"6ee98b90-1ba1-4013-bf30-8b89f1a028a6","Type":"ContainerStarted","Data":"c7752b163a70f30feef888602a8fb1b087359da4d9be1601ecdffaa4581da21f"}
Jul 28 16:21:34.579399 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.579367    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfzgs" event={"ID":"6ee98b90-1ba1-4013-bf30-8b89f1a028a6","Type":"ContainerStarted","Data":"3fbec89d8bbd66f98448fad97e123ddd7b995efc329725fc113231301f94be96"}
Jul 28 16:21:34.581352 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.581307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6h5n9" event={"ID":"e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57","Type":"ContainerStarted","Data":"35d2b5975c6f9448e2c715ace34ba1104b70b45a90f06987592cab9c5ef512fe"}
Jul 28 16:21:34.581764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.581738    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:21:34.585032 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.585012    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-l228b"
Jul 28 16:21:34.592414 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.592341    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-l228b" podStartSLOduration=2.205315713 podStartE2EDuration="24.592324554s" totalImagesPullingTime="22.387008841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:34.590510836 +0000 UTC m=+79.983827379" watchObservedRunningTime="2026-07-28 16:21:34.592324554 +0000 UTC m=+79.985641095"
Jul 28 16:21:34.635112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.634910    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nfzgs" podStartSLOduration=67.07218 podStartE2EDuration="1m19.634890628s" totalImagesPullingTime="12.562710628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:34.612883758 +0000 UTC m=+80.006200299" watchObservedRunningTime="2026-07-28 16:21:34.634890628 +0000 UTC m=+80.028207173"
Jul 28 16:21:34.657509 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:34.657445    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zx225" podStartSLOduration=66.237147131 podStartE2EDuration="1m18.657404093s" totalImagesPullingTime="12.420256962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:34.656139655 +0000 UTC m=+80.049456195" watchObservedRunningTime="2026-07-28 16:21:34.657404093 +0000 UTC m=+80.050720631"
Jul 28 16:21:35.591435 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:35.591387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"}
Jul 28 16:21:35.591859 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:35.591475    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"}
Jul 28 16:21:35.591859 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:35.591509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerStarted","Data":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"}
Jul 28 16:21:35.630744 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:35.630691    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6h5n9" podStartSLOduration=67.080408036 podStartE2EDuration="1m19.63067621s" totalImagesPullingTime="12.550268174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:34.676172872 +0000 UTC m=+80.069489412" watchObservedRunningTime="2026-07-28 16:21:35.63067621 +0000 UTC m=+81.023992747"
Jul 28 16:21:35.631474 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:35.631438    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.292785631 podStartE2EDuration="17.631421988s" totalImagesPullingTime="14.314508133s" totalInitContainerRuntime="24.128224ms" isStatefulPod=false podCreationTimestamp="2026-07-28 16:21:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:21:35.623549623 +0000 UTC m=+81.016866165" watchObservedRunningTime="2026-07-28 16:21:35.631421988 +0000 UTC m=+81.024738528"
Jul 28 16:21:39.205741 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:21:39.205706    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:06.597638 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:06.597610    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6h5n9"
Jul 28 16:22:19.205270 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:19.205233    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:19.263082 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:19.263057    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:19.775195 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:19.775172    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.415045 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.415012    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:22:37.416029 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.415992    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="prometheus" containerID="cri-o://d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e" gracePeriod=600
Jul 28 16:22:37.416256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.416001    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a" gracePeriod=600
Jul 28 16:22:37.416362 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.416039    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="thanos-sidecar" containerID="cri-o://e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321" gracePeriod=600
Jul 28 16:22:37.416466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.416000    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy" containerID="cri-o://0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3" gracePeriod=600
Jul 28 16:22:37.416466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.416054    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-web" containerID="cri-o://0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1" gracePeriod=600
Jul 28 16:22:37.416574 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.416064    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="config-reloader" containerID="cri-o://7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457" gracePeriod=600
Jul 28 16:22:37.661164 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.661137    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.781314 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781281    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config-out\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781519 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781331    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-grpc-tls\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781519 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781423    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781519 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781456    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-1\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781519 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781482    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-metrics-client-certs\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781519 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781507    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781533    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-serving-certs-ca-bundle\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781558    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-tls-assets\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781588    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-tls\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781633    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-2\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781657    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-db\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781723    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-thanos-prometheus-http-client-file\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.781757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781753    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-kubelet-serving-ca-bundle\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781779    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-trusted-ca-bundle\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781779    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-web-config\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781839    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-metrics-client-ca\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781878    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781902    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-kube-rbac-proxy\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781939    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2w5d\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-kube-api-access-g2w5d\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.781965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-0\") pod \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\" (UID: \"ec75ccde-c993-4ea8-8b4f-89e3159731f6\") "
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.782101    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.782183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.782184    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.782728 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.782201    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.782728 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.782708    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:22:37.782728 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.782713    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.783333 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.783304    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.784215 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.784190    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.784331 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.784234    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.784678 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.784614    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:37.785888 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.785860    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.786493 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.786456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.786493 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.786483    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config-out" (OuterVolumeSpecName: "config-out") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:22:37.786832 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.786807    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.786928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.786887    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.787004 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.786983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-kube-api-access-g2w5d" (OuterVolumeSpecName: "kube-api-access-g2w5d") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "kube-api-access-g2w5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:37.787109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.787090    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.787198 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.787176    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:37.787433 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.787418    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.787716 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.787704    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.787951 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.787928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config" (OuterVolumeSpecName: "config") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.796796 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.796776    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-web-config" (OuterVolumeSpecName: "web-config") pod "ec75ccde-c993-4ea8-8b4f-89e3159731f6" (UID: "ec75ccde-c993-4ea8-8b4f-89e3159731f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:37.825963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.825928    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a" exitCode=0
Jul 28 16:22:37.825963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.825958    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3" exitCode=0
Jul 28 16:22:37.825963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.825968    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1" exitCode=0
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.825976    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321" exitCode=0
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.825985    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457" exitCode=0
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.825994    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e" exitCode=0
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"}
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"}
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"}
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826169    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.826209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"}
Jul 28 16:22:37.826569 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"}
Jul 28 16:22:37.826569 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"}
Jul 28 16:22:37.826569 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826289    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec75ccde-c993-4ea8-8b4f-89e3159731f6","Type":"ContainerDied","Data":"3015ab3edd15970aeb565c5c7bb179d74bb32b35051314431430b28e2410da27"}
Jul 28 16:22:37.828788 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.826731    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.835413 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.835393    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.842448 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.842433    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.848490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.848470    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.852532 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.852511    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:22:37.855098 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.855083    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.858191 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.858173    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:22:37.861958 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.861942    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.868617 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.868597    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.874699 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.874685    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.874951 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.874933    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.874996 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.874960    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"} err="failed to get container status \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist"
Jul 28 16:22:37.874996 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.874991    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.875209 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.875194    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.875243 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875215    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"} err="failed to get container status \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist"
Jul 28 16:22:37.875243 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875230    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.875489 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.875469    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.875553 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875494    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"} err="failed to get container status \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist"
Jul 28 16:22:37.875553 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875507    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.875722 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.875705    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.875757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875727    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"} err="failed to get container status \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist"
Jul 28 16:22:37.875757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875739    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.875947 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.875929    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.875984 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875953    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"} err="failed to get container status \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist"
Jul 28 16:22:37.875984 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.875964    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.876181 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.876165    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.876235 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876186    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"} err="failed to get container status \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist"
Jul 28 16:22:37.876235 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876199    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.876409 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:22:37.876369    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.876451 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876414    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"} err="failed to get container status \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist"
Jul 28 16:22:37.876451 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876426    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.876591 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876577    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"} err="failed to get container status \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist"
Jul 28 16:22:37.876627 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876593    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.876755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876738    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"} err="failed to get container status \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist"
Jul 28 16:22:37.876793 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876755    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.876928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876914    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"} err="failed to get container status \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist"
Jul 28 16:22:37.876968 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.876929    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.877097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877083    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"} err="failed to get container status \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist"
Jul 28 16:22:37.877134 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877097    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.877271 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877255    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"} err="failed to get container status \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist"
Jul 28 16:22:37.877271 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877270    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.877463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877446    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"} err="failed to get container status \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist"
Jul 28 16:22:37.877463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877461    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.877671 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877655    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"} err="failed to get container status \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist"
Jul 28 16:22:37.877671 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877672    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.877852 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877833    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"} err="failed to get container status \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist"
Jul 28 16:22:37.877895 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.877854    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.878077 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878060    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"} err="failed to get container status \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist"
Jul 28 16:22:37.878120 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878078    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.878261 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878249    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"} err="failed to get container status \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist"
Jul 28 16:22:37.878312 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878261    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.878475 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878457    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"} err="failed to get container status \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist"
Jul 28 16:22:37.878548 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878476    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.878727 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878707    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"} err="failed to get container status \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist"
Jul 28 16:22:37.878770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878729    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.878923 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878906    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"} err="failed to get container status \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist"
Jul 28 16:22:37.878965 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.878923    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.879099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879082    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"} err="failed to get container status \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist"
Jul 28 16:22:37.879163 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879101    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.879305 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879290    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"} err="failed to get container status \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist"
Jul 28 16:22:37.879345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879305    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.879522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879508    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"} err="failed to get container status \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist"
Jul 28 16:22:37.879573 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879522    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.879685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879672    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"} err="failed to get container status \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist"
Jul 28 16:22:37.879725 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879686    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.879857 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879844    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"} err="failed to get container status \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist"
Jul 28 16:22:37.879905 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.879856    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.880031 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880015    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"} err="failed to get container status \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist"
Jul 28 16:22:37.880098 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880033    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.880236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880219    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"} err="failed to get container status \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist"
Jul 28 16:22:37.880276 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880236    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.880450 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880433    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"} err="failed to get container status \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist"
Jul 28 16:22:37.880521 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880452    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.880632 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880615    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"} err="failed to get container status \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist"
Jul 28 16:22:37.880689 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880633    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.880823 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880807    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"} err="failed to get container status \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist"
Jul 28 16:22:37.880888 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.880824    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.881033 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881017    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"} err="failed to get container status \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist"
Jul 28 16:22:37.881090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881034    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.881258 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881238    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"} err="failed to get container status \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist"
Jul 28 16:22:37.881306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881259    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.881488 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881468    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"} err="failed to get container status \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist"
Jul 28 16:22:37.881488 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881487    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.881711 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881694    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"} err="failed to get container status \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist"
Jul 28 16:22:37.881754 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881711    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.881928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881911    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"} err="failed to get container status \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist"
Jul 28 16:22:37.881996 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.881930    2573 scope.go:117] "RemoveContainer" containerID="648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"
Jul 28 16:22:37.882150 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882134    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a"} err="failed to get container status \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": rpc error: code = NotFound desc = could not find container \"648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a\": container with ID starting with 648b0141e59386b326f08fe988b29817c2e3f80376ef26524cfb793dd8a22d8a not found: ID does not exist"
Jul 28 16:22:37.882194 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882152    2573 scope.go:117] "RemoveContainer" containerID="0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"
Jul 28 16:22:37.882471 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882357    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3"} err="failed to get container status \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": rpc error: code = NotFound desc = could not find container \"0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3\": container with ID starting with 0ad0b722b739bdb2f67cc61b172228c51c357b8e0447c7b83020430e4475c6f3 not found: ID does not exist"
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882470    2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882491    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882501    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-tls-assets\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882513    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882522    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882529    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-db\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882537    2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882546    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882473    2573 scope.go:117] "RemoveContainer" containerID="0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882556    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-web-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882569    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-configmap-metrics-client-ca\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882578    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882591    2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-kube-rbac-proxy\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882601    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2w5d\" (UniqueName: \"kubernetes.io/projected/ec75ccde-c993-4ea8-8b4f-89e3159731f6-kube-api-access-g2w5d\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882608    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec75ccde-c993-4ea8-8b4f-89e3159731f6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882616    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec75ccde-c993-4ea8-8b4f-89e3159731f6-config-out\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882624    2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-grpc-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882631    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882640    2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec75ccde-c993-4ea8-8b4f-89e3159731f6-secret-metrics-client-certs\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882758    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1"} err="failed to get container status \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": rpc error: code = NotFound desc = could not find container \"0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1\": container with ID starting with 0e2602f567329e1926b3c357c264b23f3de455ea97035ef89cc7330c5465b8a1 not found: ID does not exist"
Jul 28 16:22:37.882841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882772    2573 scope.go:117] "RemoveContainer" containerID="e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"
Jul 28 16:22:37.883221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882964    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321"} err="failed to get container status \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": rpc error: code = NotFound desc = could not find container \"e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321\": container with ID starting with e4680007b5804798efe74fa05c4e2f0110d0be2444ce2dbaac4161c64a1cf321 not found: ID does not exist"
Jul 28 16:22:37.883221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.882980    2573 scope.go:117] "RemoveContainer" containerID="7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"
Jul 28 16:22:37.883221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.883170    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457"} err="failed to get container status \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": rpc error: code = NotFound desc = could not find container \"7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457\": container with ID starting with 7fada09ed08df5f5bed724d7a11fd887e6d6e84bfcb53aca5fa4e764c07aa457 not found: ID does not exist"
Jul 28 16:22:37.883221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.883201    2573 scope.go:117] "RemoveContainer" containerID="d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"
Jul 28 16:22:37.883433 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.883414    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e"} err="failed to get container status \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": rpc error: code = NotFound desc = could not find container \"d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e\": container with ID starting with d650e1d0e6a9d4493e2049b365d68120604b4a3ebce4da1cb9b86df31395656e not found: ID does not exist"
Jul 28 16:22:37.883479 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.883434    2573 scope.go:117] "RemoveContainer" containerID="bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"
Jul 28 16:22:37.883647 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.883625    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093"} err="failed to get container status \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": rpc error: code = NotFound desc = could not find container \"bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093\": container with ID starting with bf0e2afb44a043f24bb5f55c1d6a3d08197929e2326a549bab142b2bef3fa093 not found: ID does not exist"
Jul 28 16:22:37.895945 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.895927    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:22:37.896155 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896145    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="config-reloader"
Jul 28 16:22:37.896197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896158    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="config-reloader"
Jul 28 16:22:37.896197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896170    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-web"
Jul 28 16:22:37.896197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896176    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-web"
Jul 28 16:22:37.896197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896183    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-thanos"
Jul 28 16:22:37.896197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896188    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-thanos"
Jul 28 16:22:37.896197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896197    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="init-config-reloader"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896202    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="init-config-reloader"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896208    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="thanos-sidecar"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896213    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="thanos-sidecar"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896235    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896241    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896249    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="prometheus"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896254    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="prometheus"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896291    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="config-reloader"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896296    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896302    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="thanos-sidecar"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896309    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-web"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896314    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="kube-rbac-proxy-thanos"
Jul 28 16:22:37.896360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.896319    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" containerName="prometheus"
Jul 28 16:22:37.900194 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.900180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.905336 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905314    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 16:22:37.905439 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905318    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 16:22:37.905507 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905481    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 16:22:37.905582 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905567    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 16:22:37.905734 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905690    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mfc2l\""
Jul 28 16:22:37.905866 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905850    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 16:22:37.905930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905880    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 16:22:37.906099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.905988    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 16:22:37.906099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.906025    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 16:22:37.906212 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.906075    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 16:22:37.906468 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.906451    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 16:22:37.906703 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.906680    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 16:22:37.906831 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.906817    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 16:22:37.906955 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.906938    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 16:22:37.907283 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.907264    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8bat0rlh9dpan\""
Jul 28 16:22:37.911709 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.911607    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 16:22:37.913757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.913739    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 16:22:37.930205 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.930183    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:22:37.983210 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-config\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983444    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhnd\" (UniqueName: \"kubernetes.io/projected/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-kube-api-access-dvhnd\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983959 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:37.983959 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:37.983721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084239 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084239 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084239 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084286    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084355    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhnd\" (UniqueName: \"kubernetes.io/projected/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-kube-api-access-dvhnd\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-config\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084657    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.084803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.085305 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.084888    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.085305 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.085128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.085892 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.085812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087570    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087846    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087846    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.087972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.087942    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.088821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.088468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.088821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.088782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.088821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.088807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.089656 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.089635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.090056 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.090037    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.090456 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.090437    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.090545 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.090537    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-config\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.090604 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.090581    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.104963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.104942    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhnd\" (UniqueName: \"kubernetes.io/projected/0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a-kube-api-access-dvhnd\") pod \"prometheus-k8s-0\" (UID: \"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.209365 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.209339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:22:38.363022 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.362940    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 16:22:38.366638 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:22:38.366614    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe7b845_c895_4a3c_b7d3_bbef1aa9db1a.slice/crio-ce1f0f19019be5cfac53c104f65cc71f87e5c3beda8de52a8cf38e9581478d3f WatchSource:0}: Error finding container ce1f0f19019be5cfac53c104f65cc71f87e5c3beda8de52a8cf38e9581478d3f: Status 404 returned error can't find the container with id ce1f0f19019be5cfac53c104f65cc71f87e5c3beda8de52a8cf38e9581478d3f
Jul 28 16:22:38.830784 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.830751    2573 generic.go:356] "Generic (PLEG): container finished" podID="0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a" containerID="fea469562838e1ecf16800ddd08724ebce3c34a505ec73bb94269b6cbba19718" exitCode=0
Jul 28 16:22:38.831167 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.830827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerDied","Data":"fea469562838e1ecf16800ddd08724ebce3c34a505ec73bb94269b6cbba19718"}
Jul 28 16:22:38.831167 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:38.830892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"ce1f0f19019be5cfac53c104f65cc71f87e5c3beda8de52a8cf38e9581478d3f"}
Jul 28 16:22:39.174342 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.174317    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec75ccde-c993-4ea8-8b4f-89e3159731f6" path="/var/lib/kubelet/pods/ec75ccde-c993-4ea8-8b4f-89e3159731f6/volumes"
Jul 28 16:22:39.842392 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.842344    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"b6e397f2a35dc3e7068d5a8314fecce8be51b8838a10d65f16105ff025ed9715"}
Jul 28 16:22:39.842764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.842455    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"d509770aef29ec7e91c3bc1b632a1e9a2069cd58b8c186399beb3a5ba3cba927"}
Jul 28 16:22:39.842764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.842496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"07fce0374a9b9d94e17adefcd9b99aaa69649b14652f2306d1de5e786e98497d"}
Jul 28 16:22:39.842764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.842523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"4aa2c1ea2908a029f20f5ca461d50630c80a10d121ab259339ff3155db2f3e7f"}
Jul 28 16:22:39.842764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.842545    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"9e1d03427f9dcd56856d3fdf1daced4d2d88efe6ac3993a58c8bd6ddcecfdabd"}
Jul 28 16:22:39.842764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.842580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a","Type":"ContainerStarted","Data":"64411c08b99105510507cc9de4b7609884f47f66f9e47c3e49975a8b187aedac"}
Jul 28 16:22:39.872315 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:39.872274    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.852758716 podStartE2EDuration="2.872258833s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.500117ms" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:39.871061933 +0000 UTC m=+145.264378484" watchObservedRunningTime="2026-07-28 16:22:39.872258833 +0000 UTC m=+145.265575372"
Jul 28 16:22:43.209796 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:22:43.209764    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:23:38.209584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:23:38.209548    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:23:38.224866 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:23:38.224838    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:23:39.057014 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:23:39.056984    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 16:25:14.994518 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:14.994488    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:25:14.998141 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:14.998119    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:25:14.999742 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:14.999718    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 16:25:32.032804 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.032761    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-kt8h5"]
Jul 28 16:25:32.034999 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.034984    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.038462 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.038436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 16:25:32.038579 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.038473    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bk2rk\""
Jul 28 16:25:32.038579 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.038438    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 16:25:32.038579 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.038440    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 16:25:32.050724 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.050707    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-kt8h5"]
Jul 28 16:25:32.176588 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.176570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5kc\" (UniqueName: \"kubernetes.io/projected/6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a-kube-api-access-6c5kc\") pod \"seaweedfs-64568bcd49-kt8h5\" (UID: \"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a\") " pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.176691 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.176609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a-data\") pod \"seaweedfs-64568bcd49-kt8h5\" (UID: \"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a\") " pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.277109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.277076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a-data\") pod \"seaweedfs-64568bcd49-kt8h5\" (UID: \"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a\") " pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.277207 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.277145    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5kc\" (UniqueName: \"kubernetes.io/projected/6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a-kube-api-access-6c5kc\") pod \"seaweedfs-64568bcd49-kt8h5\" (UID: \"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a\") " pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.277431 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.277416    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a-data\") pod \"seaweedfs-64568bcd49-kt8h5\" (UID: \"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a\") " pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.286046 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.286003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5kc\" (UniqueName: \"kubernetes.io/projected/6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a-kube-api-access-6c5kc\") pod \"seaweedfs-64568bcd49-kt8h5\" (UID: \"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a\") " pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.343270 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.343251    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:32.469268 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.469241    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-kt8h5"]
Jul 28 16:25:32.471942 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:25:32.471905    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb7d36e_8fca_4fce_ba27_31e8e4c57b4a.slice/crio-8481a2a4ab4e50d1aeb05ea1394621efe3114db920613d095d2677f98b564a97 WatchSource:0}: Error finding container 8481a2a4ab4e50d1aeb05ea1394621efe3114db920613d095d2677f98b564a97: Status 404 returned error can't find the container with id 8481a2a4ab4e50d1aeb05ea1394621efe3114db920613d095d2677f98b564a97
Jul 28 16:25:32.473401 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:32.473363    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:25:33.418092 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:33.418057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-kt8h5" event={"ID":"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a","Type":"ContainerStarted","Data":"8481a2a4ab4e50d1aeb05ea1394621efe3114db920613d095d2677f98b564a97"}
Jul 28 16:25:36.429787 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:36.429754    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-kt8h5" event={"ID":"6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a","Type":"ContainerStarted","Data":"906793ef8e10674dcb91362ce413721035bca1addb6e1cbbbd2cb24801fe1392"}
Jul 28 16:25:36.430157 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:36.429843    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:25:36.457207 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:36.457156    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-kt8h5" podStartSLOduration=1.326535566 podStartE2EDuration="4.457144233s" totalImagesPullingTime="3.130608667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:25:36.45417534 +0000 UTC m=+321.847491878" watchObservedRunningTime="2026-07-28 16:25:36.457144233 +0000 UTC m=+321.850460771"
Jul 28 16:25:42.435486 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:25:42.435453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-kt8h5"
Jul 28 16:26:49.444139 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.444104    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-c6rqv"]
Jul 28 16:26:49.447444 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.447423    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.450845 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.450824    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8dp7l\""
Jul 28 16:26:49.450982 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.450961    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 16:26:49.459400 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.459363    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-s7ln6"]
Jul 28 16:26:49.462313 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.462293    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-c6rqv"]
Jul 28 16:26:49.462427 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.462411    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:49.464528 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.464507    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 16:26:49.464742 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.464730    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jqnsq\""
Jul 28 16:26:49.472201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.472183    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-s7ln6"]
Jul 28 16:26:49.592443 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.592422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e68a70-7e21-449f-b40d-0117cff07ee6-cert\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:49.592563 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.592455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c02573-14b6-48ff-8e3a-e725561fcefe-tls-certs\") pod \"model-serving-api-7b954ffc95-c6rqv\" (UID: \"87c02573-14b6-48ff-8e3a-e725561fcefe\") " pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.592563 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.592493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtwc\" (UniqueName: \"kubernetes.io/projected/c1e68a70-7e21-449f-b40d-0117cff07ee6-kube-api-access-zbtwc\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:49.592563 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.592550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpd28\" (UniqueName: \"kubernetes.io/projected/87c02573-14b6-48ff-8e3a-e725561fcefe-kube-api-access-zpd28\") pod \"model-serving-api-7b954ffc95-c6rqv\" (UID: \"87c02573-14b6-48ff-8e3a-e725561fcefe\") " pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.693272 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.693251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c02573-14b6-48ff-8e3a-e725561fcefe-tls-certs\") pod \"model-serving-api-7b954ffc95-c6rqv\" (UID: \"87c02573-14b6-48ff-8e3a-e725561fcefe\") " pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.693395 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.693296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtwc\" (UniqueName: \"kubernetes.io/projected/c1e68a70-7e21-449f-b40d-0117cff07ee6-kube-api-access-zbtwc\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:49.693395 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.693316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpd28\" (UniqueName: \"kubernetes.io/projected/87c02573-14b6-48ff-8e3a-e725561fcefe-kube-api-access-zpd28\") pod \"model-serving-api-7b954ffc95-c6rqv\" (UID: \"87c02573-14b6-48ff-8e3a-e725561fcefe\") " pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.693395 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.693349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e68a70-7e21-449f-b40d-0117cff07ee6-cert\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:49.693546 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:26:49.693467    2573 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 16:26:49.693546 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:26:49.693533    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1e68a70-7e21-449f-b40d-0117cff07ee6-cert podName:c1e68a70-7e21-449f-b40d-0117cff07ee6 nodeName:}" failed. No retries permitted until 2026-07-28 16:26:50.193514105 +0000 UTC m=+395.586830639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c1e68a70-7e21-449f-b40d-0117cff07ee6-cert") pod "odh-model-controller-75d9cdbddf-s7ln6" (UID: "c1e68a70-7e21-449f-b40d-0117cff07ee6") : secret "odh-model-controller-webhook-cert" not found
Jul 28 16:26:49.695858 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.695816    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c02573-14b6-48ff-8e3a-e725561fcefe-tls-certs\") pod \"model-serving-api-7b954ffc95-c6rqv\" (UID: \"87c02573-14b6-48ff-8e3a-e725561fcefe\") " pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.709297 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.709265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtwc\" (UniqueName: \"kubernetes.io/projected/c1e68a70-7e21-449f-b40d-0117cff07ee6-kube-api-access-zbtwc\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:49.711406 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.711366    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpd28\" (UniqueName: \"kubernetes.io/projected/87c02573-14b6-48ff-8e3a-e725561fcefe-kube-api-access-zpd28\") pod \"model-serving-api-7b954ffc95-c6rqv\" (UID: \"87c02573-14b6-48ff-8e3a-e725561fcefe\") " pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.757003 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.756980    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:49.888167 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:49.888134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-c6rqv"]
Jul 28 16:26:49.891540 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:26:49.891515    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c02573_14b6_48ff_8e3a_e725561fcefe.slice/crio-ad113540e6476b37a81dce4373a0ea136163680b72133950c92ad87236e3d711 WatchSource:0}: Error finding container ad113540e6476b37a81dce4373a0ea136163680b72133950c92ad87236e3d711: Status 404 returned error can't find the container with id ad113540e6476b37a81dce4373a0ea136163680b72133950c92ad87236e3d711
Jul 28 16:26:50.197992 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:50.197970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e68a70-7e21-449f-b40d-0117cff07ee6-cert\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:50.200195 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:50.200179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1e68a70-7e21-449f-b40d-0117cff07ee6-cert\") pod \"odh-model-controller-75d9cdbddf-s7ln6\" (UID: \"c1e68a70-7e21-449f-b40d-0117cff07ee6\") " pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:50.371243 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:50.371221    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:50.536410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:50.536285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-s7ln6"]
Jul 28 16:26:50.536410 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:26:50.536364    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e68a70_7e21_449f_b40d_0117cff07ee6.slice/crio-98a2126404daf1665e52625d10c7d7a990723bd90bfd435c0bb7990bbfcd4fef WatchSource:0}: Error finding container 98a2126404daf1665e52625d10c7d7a990723bd90bfd435c0bb7990bbfcd4fef: Status 404 returned error can't find the container with id 98a2126404daf1665e52625d10c7d7a990723bd90bfd435c0bb7990bbfcd4fef
Jul 28 16:26:50.675018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:50.674983    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-c6rqv" event={"ID":"87c02573-14b6-48ff-8e3a-e725561fcefe","Type":"ContainerStarted","Data":"ad113540e6476b37a81dce4373a0ea136163680b72133950c92ad87236e3d711"}
Jul 28 16:26:50.676238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:50.676210    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6" event={"ID":"c1e68a70-7e21-449f-b40d-0117cff07ee6","Type":"ContainerStarted","Data":"98a2126404daf1665e52625d10c7d7a990723bd90bfd435c0bb7990bbfcd4fef"}
Jul 28 16:26:53.692302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:53.692217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-c6rqv" event={"ID":"87c02573-14b6-48ff-8e3a-e725561fcefe","Type":"ContainerStarted","Data":"d8ba62e3c5032e69a500834e712ae3229bd050c8711ff3ae00b4e83d7c01f387"}
Jul 28 16:26:53.692736 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:53.692303    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:26:53.693675 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:53.693652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6" event={"ID":"c1e68a70-7e21-449f-b40d-0117cff07ee6","Type":"ContainerStarted","Data":"d1b6fbc3fabf82fed1ba14abc443b8e656205963292f2297829c7c9d6f4613a4"}
Jul 28 16:26:53.693815 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:53.693795    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:26:53.719042 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:26:53.719003    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-c6rqv" podStartSLOduration=1.206861731 podStartE2EDuration="4.718992688s" totalImagesPullingTime="3.512130957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:26:53.714661215 +0000 UTC m=+399.107977754" watchObservedRunningTime="2026-07-28 16:26:53.718992688 +0000 UTC m=+399.112309226"
Jul 28 16:27:04.699547 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:04.699518    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6"
Jul 28 16:27:04.701500 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:04.701479    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-c6rqv"
Jul 28 16:27:04.717755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:04.717707    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-s7ln6" podStartSLOduration=12.807556496 podStartE2EDuration="15.717694999s" totalImagesPullingTime="2.910138503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:26:53.740051661 +0000 UTC m=+399.133368201" watchObservedRunningTime="2026-07-28 16:27:04.717694999 +0000 UTC m=+410.111011516"
Jul 28 16:27:05.651636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.651603    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-tc5sd"]
Jul 28 16:27:05.654660 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.654639    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.662510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.662490    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-tc5sd"]
Jul 28 16:27:05.813347 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.813319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be88d655-6335-4d8d-b1cb-9ae7c96929cc-model-cache\") pod \"s3-init-tc5sd\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") " pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.813758 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.813429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnz2\" (UniqueName: \"kubernetes.io/projected/be88d655-6335-4d8d-b1cb-9ae7c96929cc-kube-api-access-8hnz2\") pod \"s3-init-tc5sd\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") " pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.914036 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.913977    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnz2\" (UniqueName: \"kubernetes.io/projected/be88d655-6335-4d8d-b1cb-9ae7c96929cc-kube-api-access-8hnz2\") pod \"s3-init-tc5sd\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") " pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.914036 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.914013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be88d655-6335-4d8d-b1cb-9ae7c96929cc-model-cache\") pod \"s3-init-tc5sd\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") " pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.914295 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.914282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be88d655-6335-4d8d-b1cb-9ae7c96929cc-model-cache\") pod \"s3-init-tc5sd\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") " pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.923787 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.923765    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnz2\" (UniqueName: \"kubernetes.io/projected/be88d655-6335-4d8d-b1cb-9ae7c96929cc-kube-api-access-8hnz2\") pod \"s3-init-tc5sd\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") " pod="kserve/s3-init-tc5sd"
Jul 28 16:27:05.972887 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:05.972866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tc5sd"
Jul 28 16:27:06.099267 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:06.099237    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-tc5sd"]
Jul 28 16:27:06.102359 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:27:06.102331    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe88d655_6335_4d8d_b1cb_9ae7c96929cc.slice/crio-c2ad034cd9cb6f208317067a046820c49f8452c5d85aa667fc81899ee417eb12 WatchSource:0}: Error finding container c2ad034cd9cb6f208317067a046820c49f8452c5d85aa667fc81899ee417eb12: Status 404 returned error can't find the container with id c2ad034cd9cb6f208317067a046820c49f8452c5d85aa667fc81899ee417eb12
Jul 28 16:27:06.743307 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:06.743266    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tc5sd" event={"ID":"be88d655-6335-4d8d-b1cb-9ae7c96929cc","Type":"ContainerStarted","Data":"c2ad034cd9cb6f208317067a046820c49f8452c5d85aa667fc81899ee417eb12"}
Jul 28 16:27:09.756461 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:09.756432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tc5sd" event={"ID":"be88d655-6335-4d8d-b1cb-9ae7c96929cc","Type":"ContainerStarted","Data":"db91f877338053efba088cdae73dbf4a881996e2248950c41933850ba2b28c10"}
Jul 28 16:27:12.768919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:12.768887    2573 generic.go:356] "Generic (PLEG): container finished" podID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerID="db91f877338053efba088cdae73dbf4a881996e2248950c41933850ba2b28c10" exitCode=0
Jul 28 16:27:12.769281 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:12.768954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tc5sd" event={"ID":"be88d655-6335-4d8d-b1cb-9ae7c96929cc","Type":"ContainerDied","Data":"db91f877338053efba088cdae73dbf4a881996e2248950c41933850ba2b28c10"}
Jul 28 16:27:17.793165 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:27:17.793131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tc5sd" event={"ID":"be88d655-6335-4d8d-b1cb-9ae7c96929cc","Type":"ContainerStarted","Data":"d6ce40af2f21bf24b9cf1df5f92b4ac497885641f564ea320a25b134fe41b256"}
Jul 28 16:28:22.023707 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:22.023674    2573 generic.go:356] "Generic (PLEG): container finished" podID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerID="d6ce40af2f21bf24b9cf1df5f92b4ac497885641f564ea320a25b134fe41b256" exitCode=0
Jul 28 16:28:22.024160 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:22.023747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tc5sd" event={"ID":"be88d655-6335-4d8d-b1cb-9ae7c96929cc","Type":"ContainerDied","Data":"d6ce40af2f21bf24b9cf1df5f92b4ac497885641f564ea320a25b134fe41b256"}
Jul 28 16:28:23.151986 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.151966    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tc5sd"
Jul 28 16:28:23.236797 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.236775    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be88d655-6335-4d8d-b1cb-9ae7c96929cc-model-cache\") pod \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") "
Jul 28 16:28:23.236917 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.236841    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hnz2\" (UniqueName: \"kubernetes.io/projected/be88d655-6335-4d8d-b1cb-9ae7c96929cc-kube-api-access-8hnz2\") pod \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\" (UID: \"be88d655-6335-4d8d-b1cb-9ae7c96929cc\") "
Jul 28 16:28:23.238849 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.238822    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be88d655-6335-4d8d-b1cb-9ae7c96929cc-kube-api-access-8hnz2" (OuterVolumeSpecName: "kube-api-access-8hnz2") pod "be88d655-6335-4d8d-b1cb-9ae7c96929cc" (UID: "be88d655-6335-4d8d-b1cb-9ae7c96929cc"). InnerVolumeSpecName "kube-api-access-8hnz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:28:23.258324 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.258302    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be88d655-6335-4d8d-b1cb-9ae7c96929cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "be88d655-6335-4d8d-b1cb-9ae7c96929cc" (UID: "be88d655-6335-4d8d-b1cb-9ae7c96929cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:28:23.338022 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.337964    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hnz2\" (UniqueName: \"kubernetes.io/projected/be88d655-6335-4d8d-b1cb-9ae7c96929cc-kube-api-access-8hnz2\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:28:23.338022 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:23.337986    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be88d655-6335-4d8d-b1cb-9ae7c96929cc-model-cache\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:28:24.032046 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:24.032016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-tc5sd" event={"ID":"be88d655-6335-4d8d-b1cb-9ae7c96929cc","Type":"ContainerDied","Data":"c2ad034cd9cb6f208317067a046820c49f8452c5d85aa667fc81899ee417eb12"}
Jul 28 16:28:24.032046 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:24.032050    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2ad034cd9cb6f208317067a046820c49f8452c5d85aa667fc81899ee417eb12"
Jul 28 16:28:24.032252 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:24.032051    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-tc5sd"
Jul 28 16:28:36.123651 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.123619    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"]
Jul 28 16:28:36.124016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.123897    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerName="download-hf-model"
Jul 28 16:28:36.124016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.123908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerName="download-hf-model"
Jul 28 16:28:36.124016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.123921    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerName="s3-init"
Jul 28 16:28:36.124016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.123926    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerName="s3-init"
Jul 28 16:28:36.124016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.123968    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be88d655-6335-4d8d-b1cb-9ae7c96929cc" containerName="s3-init"
Jul 28 16:28:36.126811 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.126793    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.129580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129396    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:28:36.129580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129430    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:28:36.129580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129440    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-965b2-kube-rbac-proxy-sar-config\""
Jul 28 16:28:36.129580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129448    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:28:36.129580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129465    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:28:36.129896 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129696    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-965b2-predictor-serving-cert\""
Jul 28 16:28:36.129896 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.129734    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:28:36.137815 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.137797    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"]
Jul 28 16:28:36.226242 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.226218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.226359 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.226250    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-proxy-tls\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.226359 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.226275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.226359 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.226299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.226534 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.226427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98jd\" (UniqueName: \"kubernetes.io/projected/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-api-access-b98jd\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.327657 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.327637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.327750 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.327673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98jd\" (UniqueName: \"kubernetes.io/projected/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-api-access-b98jd\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.327750 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.327701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.327750 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.327729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-proxy-tls\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.327890 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.327758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.328101 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.328078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.328429 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.328405    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.328517 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.328403    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.330306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.330286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-proxy-tls\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.336006 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.335987    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98jd\" (UniqueName: \"kubernetes.io/projected/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-api-access-b98jd\") pod \"isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.437354 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.437291    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:28:36.566506 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:36.566390    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"]
Jul 28 16:28:36.569096 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:28:36.569064    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4344bdb3_0c32_4e81_b860_5c8a45f8b2d8.slice/crio-f0f9e281929eb505dd14bf8c303732a80d89454a7dd0407896e16b487fb70f77 WatchSource:0}: Error finding container f0f9e281929eb505dd14bf8c303732a80d89454a7dd0407896e16b487fb70f77: Status 404 returned error can't find the container with id f0f9e281929eb505dd14bf8c303732a80d89454a7dd0407896e16b487fb70f77
Jul 28 16:28:37.078197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:37.078156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerStarted","Data":"eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab"}
Jul 28 16:28:37.078362 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:37.078225    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerStarted","Data":"f0f9e281929eb505dd14bf8c303732a80d89454a7dd0407896e16b487fb70f77"}
Jul 28 16:28:41.093914 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:41.093880    2573 generic.go:356] "Generic (PLEG): container finished" podID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerID="eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab" exitCode=0
Jul 28 16:28:41.094452 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:41.093960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerDied","Data":"eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab"}
Jul 28 16:28:55.166858 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:55.166817    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerStarted","Data":"4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840"}
Jul 28 16:28:57.180020 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:28:57.179984    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerStarted","Data":"aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82"}
Jul 28 16:29:00.197914 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:00.197873    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerStarted","Data":"e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf"}
Jul 28 16:29:00.198313 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:00.198140    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:29:00.198313 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:00.198219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:29:00.199504 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:00.199469    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:00.220151 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:00.220108    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podStartSLOduration=2.576627657 podStartE2EDuration="24.220092517s" totalImagesPullingTime="18.115074195s" totalInitContainerRuntime="3.528390665s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:28:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:29:00.219247574 +0000 UTC m=+525.612564115" watchObservedRunningTime="2026-07-28 16:29:00.220092517 +0000 UTC m=+525.613409056"
Jul 28 16:29:01.201640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:01.201597    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:29:01.201640 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:01.201615    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:01.202816 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:01.202791    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:01.205566 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:01.205546    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:29:02.205979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:02.205938    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:02.206419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:02.206189    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:03.210134 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:03.210098    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:03.210682 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:03.210526    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:13.210113 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:13.210020    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:13.210583 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:13.210491    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:23.210430 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:23.210393    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:23.210972 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:23.210944    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:33.210676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:33.210631    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:33.211196 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:33.211174    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:43.210588 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:43.210550    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:43.211089 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:43.211066    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:29:53.210014 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:53.209974    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:29:53.210465 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:29:53.210432    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:30:03.210592 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:03.210558    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:03.211259 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:03.211232    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:11.390563 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.390475    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"]
Jul 28 16:30:11.391197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.390850    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" containerID="cri-o://4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840" gracePeriod=30
Jul 28 16:30:11.391197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.390892    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" containerID="cri-o://e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf" gracePeriod=30
Jul 28 16:30:11.391197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.390934    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" containerID="cri-o://aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82" gracePeriod=30
Jul 28 16:30:11.392248 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.392227    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"]
Jul 28 16:30:11.395172 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.395156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.397366 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.397346    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-3b17e-predictor-serving-cert\""
Jul 28 16:30:11.397513 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.397349    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-3b17e-kube-rbac-proxy-sar-config\""
Jul 28 16:30:11.407851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.407830    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"]
Jul 28 16:30:11.516327 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.516301    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"]
Jul 28 16:30:11.519505 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.519484    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.521659 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.521644    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-3b17e-predictor-serving-cert\""
Jul 28 16:30:11.521761 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.521683    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-3b17e-kube-rbac-proxy-sar-config\""
Jul 28 16:30:11.529637 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.529619    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"]
Jul 28 16:30:11.545692 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.545674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.545786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.545722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h62p\" (UniqueName: \"kubernetes.io/projected/762283e4-6452-422c-ba38-a5c5f65f1503-kube-api-access-9h62p\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.545786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.545749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762283e4-6452-422c-ba38-a5c5f65f1503-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.545872 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.545786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-cabundle-cert\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.545872 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.545822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762283e4-6452-422c-ba38-a5c5f65f1503-proxy-tls\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647248 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f53a9b9-082f-4642-813d-50f90170d837-proxy-tls\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.647248 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h62p\" (UniqueName: \"kubernetes.io/projected/762283e4-6452-422c-ba38-a5c5f65f1503-kube-api-access-9h62p\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647248 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762283e4-6452-422c-ba38-a5c5f65f1503-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-cabundle-cert\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.647490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647297    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f53a9b9-082f-4642-813d-50f90170d837-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.647490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762283e4-6452-422c-ba38-a5c5f65f1503-proxy-tls\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.647724 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqznk\" (UniqueName: \"kubernetes.io/projected/4f53a9b9-082f-4642-813d-50f90170d837-kube-api-access-mqznk\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.647724 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647613    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762283e4-6452-422c-ba38-a5c5f65f1503-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.647918 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647901    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-cabundle-cert\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.648006 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.647990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.649765 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.649747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762283e4-6452-422c-ba38-a5c5f65f1503-proxy-tls\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.656227 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.656205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h62p\" (UniqueName: \"kubernetes.io/projected/762283e4-6452-422c-ba38-a5c5f65f1503-kube-api-access-9h62p\") pod \"isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.711596 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.711577    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:11.748181 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.748142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f53a9b9-082f-4642-813d-50f90170d837-proxy-tls\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.748315 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.748191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.748315 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.748220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f53a9b9-082f-4642-813d-50f90170d837-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.748315 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.748290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.748492 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.748324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqznk\" (UniqueName: \"kubernetes.io/projected/4f53a9b9-082f-4642-813d-50f90170d837-kube-api-access-mqznk\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.748950 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.748925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f53a9b9-082f-4642-813d-50f90170d837-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.749224 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.749206    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.749533 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.749516    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.751537 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.751514    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f53a9b9-082f-4642-813d-50f90170d837-proxy-tls\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.756983 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.756941    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqznk\" (UniqueName: \"kubernetes.io/projected/4f53a9b9-082f-4642-813d-50f90170d837-kube-api-access-mqznk\") pod \"isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.830487 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.830462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:11.841820 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.841799    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"]
Jul 28 16:30:11.843556 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:30:11.843530    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762283e4_6452_422c_ba38_a5c5f65f1503.slice/crio-35c6566327ea8eea92fa9de28642b3b148cfb0391a44bacb4eeb384a6edf22ff WatchSource:0}: Error finding container 35c6566327ea8eea92fa9de28642b3b148cfb0391a44bacb4eeb384a6edf22ff: Status 404 returned error can't find the container with id 35c6566327ea8eea92fa9de28642b3b148cfb0391a44bacb4eeb384a6edf22ff
Jul 28 16:30:11.973852 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:11.973827    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"]
Jul 28 16:30:11.976158 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:30:11.976127    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f53a9b9_082f_4642_813d_50f90170d837.slice/crio-58497cd825c6b81cd02ba4ff8d9e7f2be8d5282ee44457818ec86fbbd6b493b7 WatchSource:0}: Error finding container 58497cd825c6b81cd02ba4ff8d9e7f2be8d5282ee44457818ec86fbbd6b493b7: Status 404 returned error can't find the container with id 58497cd825c6b81cd02ba4ff8d9e7f2be8d5282ee44457818ec86fbbd6b493b7
Jul 28 16:30:12.478216 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:12.478181    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerStarted","Data":"7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072"}
Jul 28 16:30:12.478678 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:12.478248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerStarted","Data":"35c6566327ea8eea92fa9de28642b3b148cfb0391a44bacb4eeb384a6edf22ff"}
Jul 28 16:30:12.480561 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:12.480537    2573 generic.go:356] "Generic (PLEG): container finished" podID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerID="aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82" exitCode=2
Jul 28 16:30:12.480667 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:12.480605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerDied","Data":"aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82"}
Jul 28 16:30:12.482008 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:12.481982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerStarted","Data":"bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9"}
Jul 28 16:30:12.482121 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:12.482032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerStarted","Data":"58497cd825c6b81cd02ba4ff8d9e7f2be8d5282ee44457818ec86fbbd6b493b7"}
Jul 28 16:30:13.210087 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:13.210055    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:30:13.211770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:13.211735    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:30:15.020894 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:15.020865    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:30:15.022842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:15.022822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:30:15.496876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:15.496841    2573 generic.go:356] "Generic (PLEG): container finished" podID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerID="4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840" exitCode=0
Jul 28 16:30:15.496975 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:15.496872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerDied","Data":"4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840"}
Jul 28 16:30:16.202359 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:16.202321    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:30:16.501793 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:16.501766    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f53a9b9-082f-4642-813d-50f90170d837" containerID="bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9" exitCode=0
Jul 28 16:30:16.501927 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:16.501838    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerDied","Data":"bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9"}
Jul 28 16:30:16.503308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:16.503285    2573 generic.go:356] "Generic (PLEG): container finished" podID="762283e4-6452-422c-ba38-a5c5f65f1503" containerID="7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072" exitCode=0
Jul 28 16:30:16.503386 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:16.503343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerDied","Data":"7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072"}
Jul 28 16:30:17.511856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:17.511515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerStarted","Data":"19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e"}
Jul 28 16:30:17.511856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:17.511591    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerStarted","Data":"7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4"}
Jul 28 16:30:17.511856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:17.511801    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:17.511856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:17.511859    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:17.513257 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:17.513020    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:30:17.536195 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:17.536144    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podStartSLOduration=2.749973857 podStartE2EDuration="6.536126467s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.78615261s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:30:17.534139531 +0000 UTC m=+602.927456072" watchObservedRunningTime="2026-07-28 16:30:17.536126467 +0000 UTC m=+602.929443005"
Jul 28 16:30:18.516338 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:18.516296    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:30:21.202921 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:21.202872    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:30:23.210475 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:23.210437    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:30:23.212128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:23.212096    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:30:23.521436 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:23.521354    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:30:23.521980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:23.521927    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:30:26.202488 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:26.202445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:30:26.202881 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:26.202625    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:31.202934 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:31.202894    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:30:33.210729 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:33.210677    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 28 16:30:33.218922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:33.210823    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:33.218922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:33.211529    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:30:33.218922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:33.211650    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:33.522080 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:33.522039    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:30:36.202129 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.202095    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:30:36.611325 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.611292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerStarted","Data":"ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745"}
Jul 28 16:30:36.611509 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.611350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerStarted","Data":"a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523"}
Jul 28 16:30:36.611570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.611554    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:36.611626 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.611608    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:36.612934 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.612909    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:30:36.631598 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:36.631556    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podStartSLOduration=2.789696702 podStartE2EDuration="25.631542472s" totalImagesPullingTime="19.113756392s" totalInitContainerRuntime="3.728089378s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:30:36.629521016 +0000 UTC m=+622.022837555" watchObservedRunningTime="2026-07-28 16:30:36.631542472 +0000 UTC m=+622.024859010"
Jul 28 16:30:37.615549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:37.615507    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:30:41.202272 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:41.202234    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 28 16:30:42.056379 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.056338    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:42.101769 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.101747    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-rbac-proxy-sar-config\") pod \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") "
Jul 28 16:30:42.101884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.101789    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kserve-provision-location\") pod \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") "
Jul 28 16:30:42.101884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.101807    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-cabundle-cert\") pod \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") "
Jul 28 16:30:42.101884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.101839    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-proxy-tls\") pod \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") "
Jul 28 16:30:42.101884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.101864    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b98jd\" (UniqueName: \"kubernetes.io/projected/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-api-access-b98jd\") pod \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\" (UID: \"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8\") "
Jul 28 16:30:42.102102 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.102082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" (UID: "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:30:42.102205 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.102102    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" (UID: "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:30:42.102300 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.102253    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" (UID: "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:30:42.104178 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.104149    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-api-access-b98jd" (OuterVolumeSpecName: "kube-api-access-b98jd") pod "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" (UID: "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8"). InnerVolumeSpecName "kube-api-access-b98jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:30:42.104258 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.104174    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" (UID: "4344bdb3-0c32-4e81-b860-5c8a45f8b2d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:30:42.203148 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.203097    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:30:42.203148 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.203120    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:30:42.203148 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.203131    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:30:42.203148 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.203140    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:30:42.203148 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.203148    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98jd\" (UniqueName: \"kubernetes.io/projected/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8-kube-api-access-b98jd\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:30:42.626078 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.626045    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:30:42.626579 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.626546    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:30:42.637484 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.637459    2573 generic.go:356] "Generic (PLEG): container finished" podID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerID="e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf" exitCode=0
Jul 28 16:30:42.637587 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.637513    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerDied","Data":"e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf"}
Jul 28 16:30:42.637587 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.637524    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"
Jul 28 16:30:42.637587 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.637565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t" event={"ID":"4344bdb3-0c32-4e81-b860-5c8a45f8b2d8","Type":"ContainerDied","Data":"f0f9e281929eb505dd14bf8c303732a80d89454a7dd0407896e16b487fb70f77"}
Jul 28 16:30:42.637751 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.637596    2573 scope.go:117] "RemoveContainer" containerID="e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf"
Jul 28 16:30:42.645866 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.645849    2573 scope.go:117] "RemoveContainer" containerID="aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82"
Jul 28 16:30:42.653536 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.653520    2573 scope.go:117] "RemoveContainer" containerID="4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840"
Jul 28 16:30:42.660256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.660240    2573 scope.go:117] "RemoveContainer" containerID="eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab"
Jul 28 16:30:42.663336 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.663320    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"]
Jul 28 16:30:42.667143 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.667125    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-965b2-predictor-5b7d68bf6d-ksg5t"]
Jul 28 16:30:42.667948 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.667930    2573 scope.go:117] "RemoveContainer" containerID="e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf"
Jul 28 16:30:42.668219 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:30:42.668202    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf\": container with ID starting with e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf not found: ID does not exist" containerID="e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf"
Jul 28 16:30:42.668270 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668228    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf"} err="failed to get container status \"e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf\": rpc error: code = NotFound desc = could not find container \"e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf\": container with ID starting with e2621e49c655e5b2c63dbb21724431b79f1361cc3f60186ffc2e3e87ac704bcf not found: ID does not exist"
Jul 28 16:30:42.668270 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668246    2573 scope.go:117] "RemoveContainer" containerID="aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82"
Jul 28 16:30:42.668477 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:30:42.668460    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82\": container with ID starting with aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82 not found: ID does not exist" containerID="aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82"
Jul 28 16:30:42.668548 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668486    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82"} err="failed to get container status \"aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82\": rpc error: code = NotFound desc = could not find container \"aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82\": container with ID starting with aaecb249b5c8550098e0eb48fe76268fb5333495db42407f32219c774ae11d82 not found: ID does not exist"
Jul 28 16:30:42.668548 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668505    2573 scope.go:117] "RemoveContainer" containerID="4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840"
Jul 28 16:30:42.668711 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:30:42.668692    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840\": container with ID starting with 4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840 not found: ID does not exist" containerID="4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840"
Jul 28 16:30:42.668751 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668718    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840"} err="failed to get container status \"4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840\": rpc error: code = NotFound desc = could not find container \"4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840\": container with ID starting with 4b9a51879bd776965b3a345653bb100fca51d969d0ad00f2d62fb2b136268840 not found: ID does not exist"
Jul 28 16:30:42.668751 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668731    2573 scope.go:117] "RemoveContainer" containerID="eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab"
Jul 28 16:30:42.668968 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:30:42.668954    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab\": container with ID starting with eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab not found: ID does not exist" containerID="eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab"
Jul 28 16:30:42.669018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:42.668972    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab"} err="failed to get container status \"eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab\": rpc error: code = NotFound desc = could not find container \"eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab\": container with ID starting with eceede73e4a687afaa75b0e9e861b713f1ac631ccba56961e9e3f26a8ff5b0ab not found: ID does not exist"
Jul 28 16:30:43.172641 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:43.172606    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" path="/var/lib/kubelet/pods/4344bdb3-0c32-4e81-b860-5c8a45f8b2d8/volumes"
Jul 28 16:30:43.522115 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:43.522076    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:30:52.627538 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:52.627497    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:30:53.522450 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:30:53.522414    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:31:02.626973 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:02.626932    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:31:03.522444 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:03.522408    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:31:12.626515 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:12.626472    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:31:13.522163 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:13.522079    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:31:22.626593 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:22.626556    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:31:23.522031 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:23.521997    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 16:31:28.168856 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:28.168823    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:31:32.627518 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:32.627487    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:31:51.612058 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.612020    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"]
Jul 28 16:31:51.612605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.612231    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container" containerID="cri-o://7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4" gracePeriod=30
Jul 28 16:31:51.612605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.612300    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kube-rbac-proxy" containerID="cri-o://19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e" gracePeriod=30
Jul 28 16:31:51.680397 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.680337    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"]
Jul 28 16:31:51.681477 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.681454    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="storage-initializer"
Jul 28 16:31:51.681628 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.681614    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="storage-initializer"
Jul 28 16:31:51.681809 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.681793    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy"
Jul 28 16:31:51.681908 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.681897    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy"
Jul 28 16:31:51.682007 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.681995    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container"
Jul 28 16:31:51.682083 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.682073    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container"
Jul 28 16:31:51.682177 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.682166    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent"
Jul 28 16:31:51.682256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.682246    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent"
Jul 28 16:31:51.682547 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.682530    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kserve-container"
Jul 28 16:31:51.682662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.682647    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="kube-rbac-proxy"
Jul 28 16:31:51.682746 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.682735    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4344bdb3-0c32-4e81-b860-5c8a45f8b2d8" containerName="agent"
Jul 28 16:31:51.686926 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.686902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.689271 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.689251    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-0983c-predictor-serving-cert\""
Jul 28 16:31:51.689652 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.689625    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-0983c-kube-rbac-proxy-sar-config\""
Jul 28 16:31:51.699478 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.699455    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"]
Jul 28 16:31:51.785222 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.785191    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"]
Jul 28 16:31:51.787953 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.787933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:51.790074 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.790056    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-0983c-predictor-serving-cert\""
Jul 28 16:31:51.790268 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.790247    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-0983c-kube-rbac-proxy-sar-config\""
Jul 28 16:31:51.803650 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.803628    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"]
Jul 28 16:31:51.804294 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.804274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.804419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.804330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.804489 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.804432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.804489 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.804472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqh5\" (UniqueName: \"kubernetes.io/projected/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-api-access-tdqh5\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.804590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.804530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.902596 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.902522    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"]
Jul 28 16:31:51.902805 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.902760    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" containerID="cri-o://a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523" gracePeriod=30
Jul 28 16:31:51.902895 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.902772    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kube-rbac-proxy" containerID="cri-o://ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745" gracePeriod=30
Jul 28 16:31:51.905766 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.905720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.905877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.905799    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.905877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.905830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:51.905983 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.905944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqh5\" (UniqueName: \"kubernetes.io/projected/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-api-access-tdqh5\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.906035 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:51.906091 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.906193 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8kd\" (UniqueName: \"kubernetes.io/projected/133fafda-789d-47b5-99a4-f7397419f336-kube-api-access-kn8kd\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:51.906252 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133fafda-789d-47b5-99a4-f7397419f336-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:51.906306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.906306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906286    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:51.906433 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906417    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.906488 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:51.906435    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-0983c-predictor-serving-cert" not found
Jul 28 16:31:51.906533 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:51.906490    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls podName:d9638e0d-a4c7-47aa-8730-101a5ea12ffb nodeName:}" failed. No retries permitted until 2026-07-28 16:31:52.406470359 +0000 UTC m=+697.799786876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" (UID: "d9638e0d-a4c7-47aa-8730-101a5ea12ffb") : secret "isvc-sklearn-graph-raw-hpa-0983c-predictor-serving-cert" not found
Jul 28 16:31:51.906533 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.906644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.906562    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:51.913551 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.913525    2573 generic.go:356] "Generic (PLEG): container finished" podID="762283e4-6452-422c-ba38-a5c5f65f1503" containerID="19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e" exitCode=2
Jul 28 16:31:51.913645 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.913597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerDied","Data":"19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e"}
Jul 28 16:31:51.914415 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:51.914359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqh5\" (UniqueName: \"kubernetes.io/projected/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-api-access-tdqh5\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:52.007756 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.007722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.007980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.007962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.008115 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.008102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8kd\" (UniqueName: \"kubernetes.io/projected/133fafda-789d-47b5-99a4-f7397419f336-kube-api-access-kn8kd\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.008208 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.008196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133fafda-789d-47b5-99a4-f7397419f336-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.008336 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.008324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.009220 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.009136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.009477 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:52.009446    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-0983c-predictor-serving-cert" not found
Jul 28 16:31:52.009576 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:52.009522    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls podName:133fafda-789d-47b5-99a4-f7397419f336 nodeName:}" failed. No retries permitted until 2026-07-28 16:31:52.509504298 +0000 UTC m=+697.902820830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" (UID: "133fafda-789d-47b5-99a4-f7397419f336") : secret "isvc-xgboost-graph-raw-hpa-0983c-predictor-serving-cert" not found
Jul 28 16:31:52.010109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.010091    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133fafda-789d-47b5-99a4-f7397419f336-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.010270 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.010236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.020432 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.020402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8kd\" (UniqueName: \"kubernetes.io/projected/133fafda-789d-47b5-99a4-f7397419f336-kube-api-access-kn8kd\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.411477 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.411443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:52.413842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.413816    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:52.512335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.512313    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.514600 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.514576    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.599163 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.599130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:52.616345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.616315    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 16:31:52.627418 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.627392    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 16:31:52.698041 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.698009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:52.725258 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.725238    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"]
Jul 28 16:31:52.728241 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:31:52.728210    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9638e0d_a4c7_47aa_8730_101a5ea12ffb.slice/crio-acae73bf6b80eae97fae34c3c332f957354690ee0d609a27b1b274ff60c1abb9 WatchSource:0}: Error finding container acae73bf6b80eae97fae34c3c332f957354690ee0d609a27b1b274ff60c1abb9: Status 404 returned error can't find the container with id acae73bf6b80eae97fae34c3c332f957354690ee0d609a27b1b274ff60c1abb9
Jul 28 16:31:52.831357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.831335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"]
Jul 28 16:31:52.834248 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:31:52.834225    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133fafda_789d_47b5_99a4_f7397419f336.slice/crio-2fdc68a0235ed0e330c83245d2650c6b5e3de12a1baf3f5abe1746949df18510 WatchSource:0}: Error finding container 2fdc68a0235ed0e330c83245d2650c6b5e3de12a1baf3f5abe1746949df18510: Status 404 returned error can't find the container with id 2fdc68a0235ed0e330c83245d2650c6b5e3de12a1baf3f5abe1746949df18510
Jul 28 16:31:52.919730 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.919678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerStarted","Data":"d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2"}
Jul 28 16:31:52.919926 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.919748    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerStarted","Data":"2fdc68a0235ed0e330c83245d2650c6b5e3de12a1baf3f5abe1746949df18510"}
Jul 28 16:31:52.922027 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.922002    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f53a9b9-082f-4642-813d-50f90170d837" containerID="ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745" exitCode=2
Jul 28 16:31:52.922143 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.922075    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerDied","Data":"ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745"}
Jul 28 16:31:52.923713 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.923678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerStarted","Data":"e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88"}
Jul 28 16:31:52.923808 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:52.923737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerStarted","Data":"acae73bf6b80eae97fae34c3c332f957354690ee0d609a27b1b274ff60c1abb9"}
Jul 28 16:31:53.517551 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:53.517513    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 28 16:31:55.748567 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.748543    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:31:55.838508 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.838479    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqznk\" (UniqueName: \"kubernetes.io/projected/4f53a9b9-082f-4642-813d-50f90170d837-kube-api-access-mqznk\") pod \"4f53a9b9-082f-4642-813d-50f90170d837\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") "
Jul 28 16:31:55.838681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.838520    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f53a9b9-082f-4642-813d-50f90170d837-kserve-provision-location\") pod \"4f53a9b9-082f-4642-813d-50f90170d837\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") "
Jul 28 16:31:55.838681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.838577    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f53a9b9-082f-4642-813d-50f90170d837-proxy-tls\") pod \"4f53a9b9-082f-4642-813d-50f90170d837\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") "
Jul 28 16:31:55.838681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.838609    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-kube-rbac-proxy-sar-config\") pod \"4f53a9b9-082f-4642-813d-50f90170d837\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") "
Jul 28 16:31:55.838681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.838650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-cabundle-cert\") pod \"4f53a9b9-082f-4642-813d-50f90170d837\" (UID: \"4f53a9b9-082f-4642-813d-50f90170d837\") "
Jul 28 16:31:55.839001 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.838912    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f53a9b9-082f-4642-813d-50f90170d837-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f53a9b9-082f-4642-813d-50f90170d837" (UID: "4f53a9b9-082f-4642-813d-50f90170d837"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:31:55.839094 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.839012    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f53a9b9-082f-4642-813d-50f90170d837" (UID: "4f53a9b9-082f-4642-813d-50f90170d837"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:55.839094 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.839067    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f53a9b9-082f-4642-813d-50f90170d837" (UID: "4f53a9b9-082f-4642-813d-50f90170d837"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:55.840602 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.840582    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f53a9b9-082f-4642-813d-50f90170d837-kube-api-access-mqznk" (OuterVolumeSpecName: "kube-api-access-mqznk") pod "4f53a9b9-082f-4642-813d-50f90170d837" (UID: "4f53a9b9-082f-4642-813d-50f90170d837"). InnerVolumeSpecName "kube-api-access-mqznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:31:55.840719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.840704    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f53a9b9-082f-4642-813d-50f90170d837-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f53a9b9-082f-4642-813d-50f90170d837" (UID: "4f53a9b9-082f-4642-813d-50f90170d837"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:31:55.936979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.936892    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f53a9b9-082f-4642-813d-50f90170d837" containerID="a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523" exitCode=0
Jul 28 16:31:55.936979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.936946    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"
Jul 28 16:31:55.937201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.936970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerDied","Data":"a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523"}
Jul 28 16:31:55.937201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.937073    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b" event={"ID":"4f53a9b9-082f-4642-813d-50f90170d837","Type":"ContainerDied","Data":"58497cd825c6b81cd02ba4ff8d9e7f2be8d5282ee44457818ec86fbbd6b493b7"}
Jul 28 16:31:55.937201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.937112    2573 scope.go:117] "RemoveContainer" containerID="ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745"
Jul 28 16:31:55.939241 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.939223    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqznk\" (UniqueName: \"kubernetes.io/projected/4f53a9b9-082f-4642-813d-50f90170d837-kube-api-access-mqznk\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:55.939341 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.939245    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f53a9b9-082f-4642-813d-50f90170d837-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:55.939341 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.939260    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f53a9b9-082f-4642-813d-50f90170d837-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:55.939341 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.939272    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:55.939341 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.939284    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f53a9b9-082f-4642-813d-50f90170d837-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:55.945667 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.945652    2573 scope.go:117] "RemoveContainer" containerID="a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523"
Jul 28 16:31:55.952740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.952726    2573 scope.go:117] "RemoveContainer" containerID="bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9"
Jul 28 16:31:55.957747 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.957723    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"]
Jul 28 16:31:55.960070 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.960056    2573 scope.go:117] "RemoveContainer" containerID="ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745"
Jul 28 16:31:55.960682 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:55.960411    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745\": container with ID starting with ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745 not found: ID does not exist" containerID="ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745"
Jul 28 16:31:55.960682 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.960468    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745"} err="failed to get container status \"ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745\": rpc error: code = NotFound desc = could not find container \"ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745\": container with ID starting with ed5adafee0c264b24fffa02180cc583fe69ef0967cbc0de7ce4fe3e693954745 not found: ID does not exist"
Jul 28 16:31:55.960682 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.960494    2573 scope.go:117] "RemoveContainer" containerID="a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523"
Jul 28 16:31:55.960926 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:55.960784    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523\": container with ID starting with a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523 not found: ID does not exist" containerID="a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523"
Jul 28 16:31:55.960926 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.960815    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523"} err="failed to get container status \"a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523\": rpc error: code = NotFound desc = could not find container \"a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523\": container with ID starting with a1a2263feaa739b8744775c8224f8debcdafbef2874a6cb1a1103563bf495523 not found: ID does not exist"
Jul 28 16:31:55.960926 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.960838    2573 scope.go:117] "RemoveContainer" containerID="bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9"
Jul 28 16:31:55.961130 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:55.961114    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9\": container with ID starting with bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9 not found: ID does not exist" containerID="bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9"
Jul 28 16:31:55.961169 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.961136    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9"} err="failed to get container status \"bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9\": rpc error: code = NotFound desc = could not find container \"bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9\": container with ID starting with bee516d5067d5bfbfd3806819d60eb7a16a5469eaf0ca87f914311227e357ed9 not found: ID does not exist"
Jul 28 16:31:55.962271 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:55.962254    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3b17e-predictor-86bddfdd97-s7m4b"]
Jul 28 16:31:56.656557 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.656536    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:31:56.747817 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.747793    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762283e4-6452-422c-ba38-a5c5f65f1503-kserve-provision-location\") pod \"762283e4-6452-422c-ba38-a5c5f65f1503\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") "
Jul 28 16:31:56.747963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.747832    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762283e4-6452-422c-ba38-a5c5f65f1503-proxy-tls\") pod \"762283e4-6452-422c-ba38-a5c5f65f1503\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") "
Jul 28 16:31:56.747963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.747895    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h62p\" (UniqueName: \"kubernetes.io/projected/762283e4-6452-422c-ba38-a5c5f65f1503-kube-api-access-9h62p\") pod \"762283e4-6452-422c-ba38-a5c5f65f1503\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") "
Jul 28 16:31:56.747963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.747914    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-cabundle-cert\") pod \"762283e4-6452-422c-ba38-a5c5f65f1503\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") "
Jul 28 16:31:56.747963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.747940    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-kube-rbac-proxy-sar-config\") pod \"762283e4-6452-422c-ba38-a5c5f65f1503\" (UID: \"762283e4-6452-422c-ba38-a5c5f65f1503\") "
Jul 28 16:31:56.748146 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.748121    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762283e4-6452-422c-ba38-a5c5f65f1503-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "762283e4-6452-422c-ba38-a5c5f65f1503" (UID: "762283e4-6452-422c-ba38-a5c5f65f1503"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:31:56.748287 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.748260    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "762283e4-6452-422c-ba38-a5c5f65f1503" (UID: "762283e4-6452-422c-ba38-a5c5f65f1503"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:56.748287 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.748263    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "762283e4-6452-422c-ba38-a5c5f65f1503" (UID: "762283e4-6452-422c-ba38-a5c5f65f1503"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:56.750000 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.749980    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762283e4-6452-422c-ba38-a5c5f65f1503-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "762283e4-6452-422c-ba38-a5c5f65f1503" (UID: "762283e4-6452-422c-ba38-a5c5f65f1503"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:31:56.750335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.750002    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762283e4-6452-422c-ba38-a5c5f65f1503-kube-api-access-9h62p" (OuterVolumeSpecName: "kube-api-access-9h62p") pod "762283e4-6452-422c-ba38-a5c5f65f1503" (UID: "762283e4-6452-422c-ba38-a5c5f65f1503"). InnerVolumeSpecName "kube-api-access-9h62p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:31:56.849255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.849213    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/762283e4-6452-422c-ba38-a5c5f65f1503-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:56.849255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.849249    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h62p\" (UniqueName: \"kubernetes.io/projected/762283e4-6452-422c-ba38-a5c5f65f1503-kube-api-access-9h62p\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:56.849255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.849260    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:56.849255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.849268    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/762283e4-6452-422c-ba38-a5c5f65f1503-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:56.849487 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.849276    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/762283e4-6452-422c-ba38-a5c5f65f1503-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:31:56.942022 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.941990    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerID="e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88" exitCode=0
Jul 28 16:31:56.942186 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.942064    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerDied","Data":"e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88"}
Jul 28 16:31:56.943986 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.943963    2573 generic.go:356] "Generic (PLEG): container finished" podID="762283e4-6452-422c-ba38-a5c5f65f1503" containerID="7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4" exitCode=0
Jul 28 16:31:56.944073 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.944018    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"
Jul 28 16:31:56.944073 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.944040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerDied","Data":"7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4"}
Jul 28 16:31:56.944150 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.944093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n" event={"ID":"762283e4-6452-422c-ba38-a5c5f65f1503","Type":"ContainerDied","Data":"35c6566327ea8eea92fa9de28642b3b148cfb0391a44bacb4eeb384a6edf22ff"}
Jul 28 16:31:56.944150 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.944126    2573 scope.go:117] "RemoveContainer" containerID="19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e"
Jul 28 16:31:56.945618 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.945595    2573 generic.go:356] "Generic (PLEG): container finished" podID="133fafda-789d-47b5-99a4-f7397419f336" containerID="d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2" exitCode=0
Jul 28 16:31:56.945730 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.945680    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerDied","Data":"d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2"}
Jul 28 16:31:56.957732 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.957706    2573 scope.go:117] "RemoveContainer" containerID="7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4"
Jul 28 16:31:56.968875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.968849    2573 scope.go:117] "RemoveContainer" containerID="7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072"
Jul 28 16:31:56.980222 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.980198    2573 scope.go:117] "RemoveContainer" containerID="19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e"
Jul 28 16:31:56.980528 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:56.980506    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e\": container with ID starting with 19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e not found: ID does not exist" containerID="19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e"
Jul 28 16:31:56.980601 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.980536    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e"} err="failed to get container status \"19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e\": rpc error: code = NotFound desc = could not find container \"19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e\": container with ID starting with 19db9b1d43542157e82a76636c2208755c20648acf08f0166eb29b1f1f76ea5e not found: ID does not exist"
Jul 28 16:31:56.980601 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.980553    2573 scope.go:117] "RemoveContainer" containerID="7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4"
Jul 28 16:31:56.980812 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:56.980792    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4\": container with ID starting with 7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4 not found: ID does not exist" containerID="7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4"
Jul 28 16:31:56.980877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.980817    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4"} err="failed to get container status \"7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4\": rpc error: code = NotFound desc = could not find container \"7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4\": container with ID starting with 7ce9dc0fdbb583790c115ea72212963ba19cb829f60a26d21ed035f8034b72a4 not found: ID does not exist"
Jul 28 16:31:56.980877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.980833    2573 scope.go:117] "RemoveContainer" containerID="7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072"
Jul 28 16:31:56.981115 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:31:56.981093    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072\": container with ID starting with 7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072 not found: ID does not exist" containerID="7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072"
Jul 28 16:31:56.981170 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:56.981123    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072"} err="failed to get container status \"7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072\": rpc error: code = NotFound desc = could not find container \"7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072\": container with ID starting with 7f5f9c7a9262c11696a0a46f0372a989021c8dd80d6241fe5100ca0601815072 not found: ID does not exist"
Jul 28 16:31:57.004325 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.004306    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"]
Jul 28 16:31:57.006434 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.006412    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3b17e-predictor-6c84db749b-lpm5n"]
Jul 28 16:31:57.172766 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.172734    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f53a9b9-082f-4642-813d-50f90170d837" path="/var/lib/kubelet/pods/4f53a9b9-082f-4642-813d-50f90170d837/volumes"
Jul 28 16:31:57.173189 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.173177    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" path="/var/lib/kubelet/pods/762283e4-6452-422c-ba38-a5c5f65f1503/volumes"
Jul 28 16:31:57.952863 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.952833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerStarted","Data":"22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf"}
Jul 28 16:31:57.953296 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.952898    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerStarted","Data":"c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e"}
Jul 28 16:31:57.953296 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.953191    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:57.953442 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.953360    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:31:57.954896 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.954866    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:31:57.956076 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.956054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerStarted","Data":"b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0"}
Jul 28 16:31:57.956177 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.956108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerStarted","Data":"9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c"}
Jul 28 16:31:57.956344 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.956327    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:57.956437 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.956387    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:31:57.957257 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.957238    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:31:57.972346 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.972298    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podStartSLOduration=3.2957978470000002 podStartE2EDuration="6.97228368s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.676485833s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:31:57.97091085 +0000 UTC m=+703.364227388" watchObservedRunningTime="2026-07-28 16:31:57.97228368 +0000 UTC m=+703.365600218"
Jul 28 16:31:57.990141 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:57.990096    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podStartSLOduration=3.45651807 podStartE2EDuration="6.990083212s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.533565142s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:31:57.988783364 +0000 UTC m=+703.382099925" watchObservedRunningTime="2026-07-28 16:31:57.990083212 +0000 UTC m=+703.383399819"
Jul 28 16:31:58.960369 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:58.960336    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:31:58.960796 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:31:58.960465    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:32:03.965041 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:03.965007    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:32:03.965505 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:03.965361    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:32:03.965505 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:03.965462    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:32:03.965833 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:03.965811    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:32:13.965615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:13.965576    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:32:13.966013 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:13.965707    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:32:23.965460 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:23.965422    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:32:23.965975 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:23.965709    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:32:33.965695 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:33.965658    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:32:33.966127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:33.965810    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:32:43.966316 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:43.966274    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:32:43.966710 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:43.966274    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:32:53.965861 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:53.965817    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:32:53.966368 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:32:53.966350    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:33:03.966322 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:03.966286    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:33:13.966534 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:13.966505    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:33:41.953393 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.953302    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"]
Jul 28 16:33:41.953842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.953608    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" containerID="cri-o://c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e" gracePeriod=30
Jul 28 16:33:41.953842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.953666    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kube-rbac-proxy" containerID="cri-o://22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf" gracePeriod=30
Jul 28 16:33:41.986723 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.986694    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"]
Jul 28 16:33:41.987011 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987000    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="storage-initializer"
Jul 28 16:33:41.987053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987013    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="storage-initializer"
Jul 28 16:33:41.987053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987029    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="storage-initializer"
Jul 28 16:33:41.987053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987034    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="storage-initializer"
Jul 28 16:33:41.987053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987047    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kube-rbac-proxy"
Jul 28 16:33:41.987053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987052    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kube-rbac-proxy"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987058    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kube-rbac-proxy"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987063    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kube-rbac-proxy"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987069    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987074    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987082    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987086    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987130    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kserve-container"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987138    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f53a9b9-082f-4642-813d-50f90170d837" containerName="kube-rbac-proxy"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987144    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kube-rbac-proxy"
Jul 28 16:33:41.987197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.987150    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="762283e4-6452-422c-ba38-a5c5f65f1503" containerName="kserve-container"
Jul 28 16:33:41.989273 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.989259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:41.991591 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.991570    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9ed13-kube-rbac-proxy-sar-config\""
Jul 28 16:33:41.991681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.991619    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-9ed13-predictor-serving-cert\""
Jul 28 16:33:41.998571 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.998552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:41.998657 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.998577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzs5\" (UniqueName: \"kubernetes.io/projected/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-api-access-tvzs5\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:41.998657 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:41.998613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.001289 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.001269    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"]
Jul 28 16:33:42.096672 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.096642    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"]
Jul 28 16:33:42.096918 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.096892    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" containerID="cri-o://9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c" gracePeriod=30
Jul 28 16:33:42.097030 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.096929    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kube-rbac-proxy" containerID="cri-o://b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0" gracePeriod=30
Jul 28 16:33:42.099865 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.099834    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.099980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.099882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzs5\" (UniqueName: \"kubernetes.io/projected/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-api-access-tvzs5\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.099980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.099931    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.100085 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:42.100047    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-serving-cert: secret "message-dumper-raw-9ed13-predictor-serving-cert" not found
Jul 28 16:33:42.100128 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:42.100101    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls podName:f06b92fd-2110-4e92-bcae-c0e6bb9ff08e nodeName:}" failed. No retries permitted until 2026-07-28 16:33:42.600086969 +0000 UTC m=+807.993403485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls") pod "message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" (UID: "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e") : secret "message-dumper-raw-9ed13-predictor-serving-cert" not found
Jul 28 16:33:42.100553 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.100534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.109674 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.109653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzs5\" (UniqueName: \"kubernetes.io/projected/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-api-access-tvzs5\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.359255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.359225    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerID="22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf" exitCode=2
Jul 28 16:33:42.359437 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.359295    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerDied","Data":"22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf"}
Jul 28 16:33:42.361109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.361089    2573 generic.go:356] "Generic (PLEG): container finished" podID="133fafda-789d-47b5-99a4-f7397419f336" containerID="b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0" exitCode=2
Jul 28 16:33:42.361211 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.361129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerDied","Data":"b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0"}
Jul 28 16:33:42.604574 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.604539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.606992 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.606961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls\") pod \"message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") " pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:42.899033 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:42.898941    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:43.039357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.039328    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"]
Jul 28 16:33:43.042643 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:33:43.042618    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06b92fd_2110_4e92_bcae_c0e6bb9ff08e.slice/crio-85a6fe8a2c1420e310b2dfb324548dc8995f492de325cb9bf973ab9b613bad23 WatchSource:0}: Error finding container 85a6fe8a2c1420e310b2dfb324548dc8995f492de325cb9bf973ab9b613bad23: Status 404 returned error can't find the container with id 85a6fe8a2c1420e310b2dfb324548dc8995f492de325cb9bf973ab9b613bad23
Jul 28 16:33:43.044694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.044677    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:33:43.366053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.366016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" event={"ID":"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e","Type":"ContainerStarted","Data":"85a6fe8a2c1420e310b2dfb324548dc8995f492de325cb9bf973ab9b613bad23"}
Jul 28 16:33:43.961072 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.961035    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 28 16:33:43.961234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.961073    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 28 16:33:43.965824 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.965789    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 16:33:43.965940 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:43.965789    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 16:33:44.372821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:44.372787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" event={"ID":"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e","Type":"ContainerStarted","Data":"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"}
Jul 28 16:33:44.373236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:44.372851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" event={"ID":"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e","Type":"ContainerStarted","Data":"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"}
Jul 28 16:33:44.373236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:44.372878    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:44.373236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:44.372916    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:44.374805 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:44.374787    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:44.393615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:44.393249    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" podStartSLOduration=2.338817658 podStartE2EDuration="3.393230118s" totalImagesPullingTime="1.05441246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:33:44.392849109 +0000 UTC m=+809.786165649" watchObservedRunningTime="2026-07-28 16:33:44.393230118 +0000 UTC m=+809.786546658"
Jul 28 16:33:45.838993 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.838968    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:33:45.930903 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.930872    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-kube-rbac-proxy-sar-config\") pod \"133fafda-789d-47b5-99a4-f7397419f336\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") "
Jul 28 16:33:45.931054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.930929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133fafda-789d-47b5-99a4-f7397419f336-kserve-provision-location\") pod \"133fafda-789d-47b5-99a4-f7397419f336\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") "
Jul 28 16:33:45.931054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.930960    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-cabundle-cert\") pod \"133fafda-789d-47b5-99a4-f7397419f336\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") "
Jul 28 16:33:45.931054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.931033    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls\") pod \"133fafda-789d-47b5-99a4-f7397419f336\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") "
Jul 28 16:33:45.931209 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.931060    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn8kd\" (UniqueName: \"kubernetes.io/projected/133fafda-789d-47b5-99a4-f7397419f336-kube-api-access-kn8kd\") pod \"133fafda-789d-47b5-99a4-f7397419f336\" (UID: \"133fafda-789d-47b5-99a4-f7397419f336\") "
Jul 28 16:33:45.931307 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.931256    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "133fafda-789d-47b5-99a4-f7397419f336" (UID: "133fafda-789d-47b5-99a4-f7397419f336"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:45.931390 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.931300    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133fafda-789d-47b5-99a4-f7397419f336-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "133fafda-789d-47b5-99a4-f7397419f336" (UID: "133fafda-789d-47b5-99a4-f7397419f336"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:33:45.931453 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.931404    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "133fafda-789d-47b5-99a4-f7397419f336" (UID: "133fafda-789d-47b5-99a4-f7397419f336"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:45.933291 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.933270    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133fafda-789d-47b5-99a4-f7397419f336-kube-api-access-kn8kd" (OuterVolumeSpecName: "kube-api-access-kn8kd") pod "133fafda-789d-47b5-99a4-f7397419f336" (UID: "133fafda-789d-47b5-99a4-f7397419f336"). InnerVolumeSpecName "kube-api-access-kn8kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:33:45.933368 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:45.933304    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "133fafda-789d-47b5-99a4-f7397419f336" (UID: "133fafda-789d-47b5-99a4-f7397419f336"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:33:46.031959 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.031933    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/133fafda-789d-47b5-99a4-f7397419f336-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.031959 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.031959    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.032118 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.031972    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/133fafda-789d-47b5-99a4-f7397419f336-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.032118 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.031980    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8kd\" (UniqueName: \"kubernetes.io/projected/133fafda-789d-47b5-99a4-f7397419f336-kube-api-access-kn8kd\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.032118 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.031988    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/133fafda-789d-47b5-99a4-f7397419f336-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.384090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.383997    2573 generic.go:356] "Generic (PLEG): container finished" podID="133fafda-789d-47b5-99a4-f7397419f336" containerID="9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c" exitCode=0
Jul 28 16:33:46.384090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.384053    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"
Jul 28 16:33:46.384090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.384075    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerDied","Data":"9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c"}
Jul 28 16:33:46.384326 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.384129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk" event={"ID":"133fafda-789d-47b5-99a4-f7397419f336","Type":"ContainerDied","Data":"2fdc68a0235ed0e330c83245d2650c6b5e3de12a1baf3f5abe1746949df18510"}
Jul 28 16:33:46.384326 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.384154    2573 scope.go:117] "RemoveContainer" containerID="b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0"
Jul 28 16:33:46.392236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.392219    2573 scope.go:117] "RemoveContainer" containerID="9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c"
Jul 28 16:33:46.399363 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.399345    2573 scope.go:117] "RemoveContainer" containerID="d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2"
Jul 28 16:33:46.405915 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.405894    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"]
Jul 28 16:33:46.406748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.406734    2573 scope.go:117] "RemoveContainer" containerID="b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0"
Jul 28 16:33:46.406983 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:46.406967    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0\": container with ID starting with b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0 not found: ID does not exist" containerID="b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0"
Jul 28 16:33:46.407022 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.406990    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0"} err="failed to get container status \"b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0\": rpc error: code = NotFound desc = could not find container \"b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0\": container with ID starting with b6a15f17615c283217254f39cb9eb2b85d4fb386a4713f929288eea706d33ff0 not found: ID does not exist"
Jul 28 16:33:46.407022 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.407005    2573 scope.go:117] "RemoveContainer" containerID="9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c"
Jul 28 16:33:46.407208 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:46.407193    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c\": container with ID starting with 9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c not found: ID does not exist" containerID="9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c"
Jul 28 16:33:46.407255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.407213    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c"} err="failed to get container status \"9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c\": rpc error: code = NotFound desc = could not find container \"9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c\": container with ID starting with 9867b5b4ebb18ad4349ad29353518666a2a9ace172a197b7f1d7cd95719e9e9c not found: ID does not exist"
Jul 28 16:33:46.407255 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.407225    2573 scope.go:117] "RemoveContainer" containerID="d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2"
Jul 28 16:33:46.407446 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:46.407404    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2\": container with ID starting with d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2 not found: ID does not exist" containerID="d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2"
Jul 28 16:33:46.407446 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.407421    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2"} err="failed to get container status \"d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2\": rpc error: code = NotFound desc = could not find container \"d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2\": container with ID starting with d5f7193be1afd2530c844e78f8e334f256b89138cfd5ab95d0f6071d40f327c2 not found: ID does not exist"
Jul 28 16:33:46.410560 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.410539    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-0983c-predictor-76d6b6fdd5-gpqzk"]
Jul 28 16:33:46.688195 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.688175    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:33:46.738159 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738133    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdqh5\" (UniqueName: \"kubernetes.io/projected/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-api-access-tdqh5\") pod \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") "
Jul 28 16:33:46.738330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-cabundle-cert\") pod \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") "
Jul 28 16:33:46.738330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738231    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls\") pod \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") "
Jul 28 16:33:46.738330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738254    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-rbac-proxy-sar-config\") pod \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") "
Jul 28 16:33:46.738330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738302    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kserve-provision-location\") pod \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\" (UID: \"d9638e0d-a4c7-47aa-8730-101a5ea12ffb\") "
Jul 28 16:33:46.738594 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738566    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9638e0d-a4c7-47aa-8730-101a5ea12ffb" (UID: "d9638e0d-a4c7-47aa-8730-101a5ea12ffb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:46.738692 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738670    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9638e0d-a4c7-47aa-8730-101a5ea12ffb" (UID: "d9638e0d-a4c7-47aa-8730-101a5ea12ffb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:33:46.738748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.738697    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9638e0d-a4c7-47aa-8730-101a5ea12ffb" (UID: "d9638e0d-a4c7-47aa-8730-101a5ea12ffb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:33:46.740221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.740198    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-api-access-tdqh5" (OuterVolumeSpecName: "kube-api-access-tdqh5") pod "d9638e0d-a4c7-47aa-8730-101a5ea12ffb" (UID: "d9638e0d-a4c7-47aa-8730-101a5ea12ffb"). InnerVolumeSpecName "kube-api-access-tdqh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:33:46.740302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.740261    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9638e0d-a4c7-47aa-8730-101a5ea12ffb" (UID: "d9638e0d-a4c7-47aa-8730-101a5ea12ffb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:33:46.839756 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.839736    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.839756 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.839753    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.840049 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.839762    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.840049 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.839771    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqh5\" (UniqueName: \"kubernetes.io/projected/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-kube-api-access-tdqh5\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:46.840049 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:46.839779    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9638e0d-a4c7-47aa-8730-101a5ea12ffb-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:33:47.172520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.172478    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="133fafda-789d-47b5-99a4-f7397419f336" path="/var/lib/kubelet/pods/133fafda-789d-47b5-99a4-f7397419f336/volumes"
Jul 28 16:33:47.391219 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.391183    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerID="c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e" exitCode=0
Jul 28 16:33:47.391456 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.391257    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"
Jul 28 16:33:47.391456 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.391245    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerDied","Data":"c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e"}
Jul 28 16:33:47.391564 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.391453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs" event={"ID":"d9638e0d-a4c7-47aa-8730-101a5ea12ffb","Type":"ContainerDied","Data":"acae73bf6b80eae97fae34c3c332f957354690ee0d609a27b1b274ff60c1abb9"}
Jul 28 16:33:47.391564 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.391483    2573 scope.go:117] "RemoveContainer" containerID="22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf"
Jul 28 16:33:47.402424 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.402351    2573 scope.go:117] "RemoveContainer" containerID="c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e"
Jul 28 16:33:47.410269 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.410248    2573 scope.go:117] "RemoveContainer" containerID="e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88"
Jul 28 16:33:47.410269 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.410261    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"]
Jul 28 16:33:47.415882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.415859    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-0983c-predictor-6dc5957ff6-hmzbs"]
Jul 28 16:33:47.418591 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.418570    2573 scope.go:117] "RemoveContainer" containerID="22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf"
Jul 28 16:33:47.418857 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:47.418839    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf\": container with ID starting with 22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf not found: ID does not exist" containerID="22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf"
Jul 28 16:33:47.418927 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.418868    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf"} err="failed to get container status \"22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf\": rpc error: code = NotFound desc = could not find container \"22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf\": container with ID starting with 22079f9a7651287c9eb3ab4b0a93ba685490f7ad675f8830f0c80968950e25cf not found: ID does not exist"
Jul 28 16:33:47.418927 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.418893    2573 scope.go:117] "RemoveContainer" containerID="c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e"
Jul 28 16:33:47.419179 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:47.419163    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e\": container with ID starting with c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e not found: ID does not exist" containerID="c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e"
Jul 28 16:33:47.419245 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.419184    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e"} err="failed to get container status \"c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e\": rpc error: code = NotFound desc = could not find container \"c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e\": container with ID starting with c037cbd8fe19871de645918b4e0695d9161924319da23280139a5734fc7fea8e not found: ID does not exist"
Jul 28 16:33:47.419245 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.419202    2573 scope.go:117] "RemoveContainer" containerID="e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88"
Jul 28 16:33:47.419493 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:47.419477    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88\": container with ID starting with e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88 not found: ID does not exist" containerID="e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88"
Jul 28 16:33:47.419561 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:47.419499    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88"} err="failed to get container status \"e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88\": rpc error: code = NotFound desc = could not find container \"e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88\": container with ID starting with e298ff7fee0401b0d8127a38da9327fbc652f08bc379aaa786093f2530032d88 not found: ID does not exist"
Jul 28 16:33:49.172449 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:49.172419    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" path="/var/lib/kubelet/pods/d9638e0d-a4c7-47aa-8730-101a5ea12ffb/volumes"
Jul 28 16:33:51.389330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:51.389300    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:33:52.041281 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041250    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"]
Jul 28 16:33:52.041634 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041619    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="storage-initializer"
Jul 28 16:33:52.041685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041636    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="storage-initializer"
Jul 28 16:33:52.041685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041646    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="storage-initializer"
Jul 28 16:33:52.041685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041652    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="storage-initializer"
Jul 28 16:33:52.041685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041666    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container"
Jul 28 16:33:52.041685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041676    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container"
Jul 28 16:33:52.041685 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041685    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041701    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041709    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kube-rbac-proxy"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kube-rbac-proxy"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041723    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kube-rbac-proxy"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041729    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kube-rbac-proxy"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041801    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kube-rbac-proxy"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041810    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9638e0d-a4c7-47aa-8730-101a5ea12ffb" containerName="kserve-container"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041818    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kserve-container"
Jul 28 16:33:52.041891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.041825    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="133fafda-789d-47b5-99a4-f7397419f336" containerName="kube-rbac-proxy"
Jul 28 16:33:52.045611 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.045588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.048210 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.048171    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9ed13-predictor-serving-cert\""
Jul 28 16:33:52.048210 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.048187    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-9ed13-kube-rbac-proxy-sar-config\""
Jul 28 16:33:52.048210 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.048224    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:33:52.048618 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.048596    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:33:52.058582 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.058549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"]
Jul 28 16:33:52.079614 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.079576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggbp\" (UniqueName: \"kubernetes.io/projected/ae69928a-1e66-435e-b05f-7ca27835572e-kube-api-access-wggbp\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.079798 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.079631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-cabundle-cert\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.079798 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.079706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.079798 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.079748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.079798 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.079774    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae69928a-1e66-435e-b05f-7ca27835572e-kserve-provision-location\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181025 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.180991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggbp\" (UniqueName: \"kubernetes.io/projected/ae69928a-1e66-435e-b05f-7ca27835572e-kube-api-access-wggbp\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-cabundle-cert\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae69928a-1e66-435e-b05f-7ca27835572e-kserve-provision-location\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181462 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:52.181239    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-serving-cert: secret "isvc-logger-raw-9ed13-predictor-serving-cert" not found
Jul 28 16:33:52.181462 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:33:52.181309    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls podName:ae69928a-1e66-435e-b05f-7ca27835572e nodeName:}" failed. No retries permitted until 2026-07-28 16:33:52.681291912 +0000 UTC m=+818.074608430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls") pod "isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" (UID: "ae69928a-1e66-435e-b05f-7ca27835572e") : secret "isvc-logger-raw-9ed13-predictor-serving-cert" not found
Jul 28 16:33:52.181576 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181520    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae69928a-1e66-435e-b05f-7ca27835572e-kserve-provision-location\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.181903 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.181857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-cabundle-cert\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.193108 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.193071    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggbp\" (UniqueName: \"kubernetes.io/projected/ae69928a-1e66-435e-b05f-7ca27835572e-kube-api-access-wggbp\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.686016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.685980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.688530 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.688507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls\") pod \"isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:52.957642 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:52.957546    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:53.091663 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:53.091637    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"]
Jul 28 16:33:53.094211 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:33:53.094183    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae69928a_1e66_435e_b05f_7ca27835572e.slice/crio-0d21288b9a5e67361a44f91efccf48a63a02c9594baa444abbf7f785e9395e76 WatchSource:0}: Error finding container 0d21288b9a5e67361a44f91efccf48a63a02c9594baa444abbf7f785e9395e76: Status 404 returned error can't find the container with id 0d21288b9a5e67361a44f91efccf48a63a02c9594baa444abbf7f785e9395e76
Jul 28 16:33:53.418354 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:53.418320    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerStarted","Data":"212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718"}
Jul 28 16:33:53.418535 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:53.418407    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerStarted","Data":"0d21288b9a5e67361a44f91efccf48a63a02c9594baa444abbf7f785e9395e76"}
Jul 28 16:33:57.440323 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:57.440232    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae69928a-1e66-435e-b05f-7ca27835572e" containerID="212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718" exitCode=0
Jul 28 16:33:57.440323 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:57.440307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerDied","Data":"212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718"}
Jul 28 16:33:58.446786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.446750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerStarted","Data":"1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d"}
Jul 28 16:33:58.447186 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.446836    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerStarted","Data":"fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc"}
Jul 28 16:33:58.447186 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.446859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerStarted","Data":"a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482"}
Jul 28 16:33:58.447186 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.447163    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:58.447329 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.447221    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:58.448611 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.448587    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:33:58.470269 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:58.470218    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podStartSLOduration=2.651863917 podStartE2EDuration="6.470204999s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.818341082s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:33:58.469125747 +0000 UTC m=+823.862442316" watchObservedRunningTime="2026-07-28 16:33:58.470204999 +0000 UTC m=+823.863521537"
Jul 28 16:33:59.450367 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:59.450339    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:33:59.450777 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:59.450479    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:33:59.451610 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:33:59.451585    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:00.453900 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:00.453863    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:00.454417 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:00.454395    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:05.458634 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:05.458602    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:34:05.459142 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:05.459104    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:05.459587 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:05.459566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:15.459980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:15.459881    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:15.460468 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:15.460304    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:25.459613 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:25.459574    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:25.460096 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:25.460038    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:35.459701 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:35.459660    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:35.460153 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:35.460130    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:45.459774 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:45.459722    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:45.460277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:45.460196    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:34:55.459777 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:55.459731    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:34:55.460235 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:34:55.460213    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:05.460053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:05.459960    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:35:05.460467 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:05.460318    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:15.046318 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:15.046291    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:35:15.046771 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:15.046509    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:35:15.460471 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:15.460402    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:15.460769 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:15.460753    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:27.063591 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.063554    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2_f06b92fd-2110-4e92-bcae-c0e6bb9ff08e/kserve-container/0.log"
Jul 28 16:35:27.313832 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.313737    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"]
Jul 28 16:35:27.314035 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.314005    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" containerID="cri-o://a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482" gracePeriod=30
Jul 28 16:35:27.314173 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.314027    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" containerID="cri-o://1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d" gracePeriod=30
Jul 28 16:35:27.314173 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.314054    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" containerID="cri-o://fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc" gracePeriod=30
Jul 28 16:35:27.375616 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.375583    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"]
Jul 28 16:35:27.378088 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.378068    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.379978 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.379960    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-6d6fe-predictor-serving-cert\""
Jul 28 16:35:27.380420 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.380407    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-6d6fe-kube-rbac-proxy-sar-config\""
Jul 28 16:35:27.389932 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.389912    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"]
Jul 28 16:35:27.465277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.465249    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"]
Jul 28 16:35:27.465516 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.465492    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kserve-container" containerID="cri-o://e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed" gracePeriod=30
Jul 28 16:35:27.465615 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.465555    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kube-rbac-proxy" containerID="cri-o://f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504" gracePeriod=30
Jul 28 16:35:27.545929 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.545902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.546029 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.545953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvck\" (UniqueName: \"kubernetes.io/projected/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-api-access-5jvck\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.546029 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.546010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-proxy-tls\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.546099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.546060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.546134 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.546103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.647007 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.646979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.647128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.647044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.647128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.647067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvck\" (UniqueName: \"kubernetes.io/projected/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-api-access-5jvck\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.647128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.647092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-proxy-tls\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.647128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.647122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.647837 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.647775    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.648204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.648143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.648204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.648149    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.650589 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.650556    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-proxy-tls\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.656737 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.656711    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvck\" (UniqueName: \"kubernetes.io/projected/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-api-access-5jvck\") pod \"isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.687544 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.687514    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:27.708386 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.708349    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:35:27.803497 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.803470    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae69928a-1e66-435e-b05f-7ca27835572e" containerID="fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc" exitCode=2
Jul 28 16:35:27.803653 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.803538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerDied","Data":"fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc"}
Jul 28 16:35:27.805011 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.804989    2573 generic.go:356] "Generic (PLEG): container finished" podID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerID="f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504" exitCode=2
Jul 28 16:35:27.805011 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.805009    2573 generic.go:356] "Generic (PLEG): container finished" podID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerID="e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed" exitCode=2
Jul 28 16:35:27.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.805031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" event={"ID":"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e","Type":"ContainerDied","Data":"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"}
Jul 28 16:35:27.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.805070    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" event={"ID":"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e","Type":"ContainerDied","Data":"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"}
Jul 28 16:35:27.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.805089    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2" event={"ID":"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e","Type":"ContainerDied","Data":"85a6fe8a2c1420e310b2dfb324548dc8995f492de325cb9bf973ab9b613bad23"}
Jul 28 16:35:27.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.805103    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"
Jul 28 16:35:27.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.805112    2573 scope.go:117] "RemoveContainer" containerID="f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"
Jul 28 16:35:27.813056 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.813029    2573 scope.go:117] "RemoveContainer" containerID="e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"
Jul 28 16:35:27.820008 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.819991    2573 scope.go:117] "RemoveContainer" containerID="f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"
Jul 28 16:35:27.820253 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:27.820235    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504\": container with ID starting with f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504 not found: ID does not exist" containerID="f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"
Jul 28 16:35:27.820316 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820262    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"} err="failed to get container status \"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504\": rpc error: code = NotFound desc = could not find container \"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504\": container with ID starting with f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504 not found: ID does not exist"
Jul 28 16:35:27.820316 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820278    2573 scope.go:117] "RemoveContainer" containerID="e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"
Jul 28 16:35:27.820536 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:27.820519    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed\": container with ID starting with e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed not found: ID does not exist" containerID="e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"
Jul 28 16:35:27.820588 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820542    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"} err="failed to get container status \"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed\": rpc error: code = NotFound desc = could not find container \"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed\": container with ID starting with e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed not found: ID does not exist"
Jul 28 16:35:27.820588 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820556    2573 scope.go:117] "RemoveContainer" containerID="f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"
Jul 28 16:35:27.820782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820762    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504"} err="failed to get container status \"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504\": rpc error: code = NotFound desc = could not find container \"f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504\": container with ID starting with f72b8b0e41455ee17286913f3dd7e9a91de50a9b7c20ee578f308f6a647e0504 not found: ID does not exist"
Jul 28 16:35:27.820826 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820784    2573 scope.go:117] "RemoveContainer" containerID="e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"
Jul 28 16:35:27.821010 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.820995    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed"} err="failed to get container status \"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed\": rpc error: code = NotFound desc = could not find container \"e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed\": container with ID starting with e78a8fa18812cd31d4e51abb68aeb51e959f72c644b39457b52cb700c4d0caed not found: ID does not exist"
Jul 28 16:35:27.829263 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.829218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"]
Jul 28 16:35:27.832128 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:35:27.832112    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b9b87e_bfbd_48dc_8208_e927dd60e1d3.slice/crio-a046ca11ec948c949699637cea8d8e144b837c664f7027691f60a0f84fe10e49 WatchSource:0}: Error finding container a046ca11ec948c949699637cea8d8e144b837c664f7027691f60a0f84fe10e49: Status 404 returned error can't find the container with id a046ca11ec948c949699637cea8d8e144b837c664f7027691f60a0f84fe10e49
Jul 28 16:35:27.848368 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.848346    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-rbac-proxy-sar-config\") pod \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") "
Jul 28 16:35:27.848491 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.848404    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls\") pod \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") "
Jul 28 16:35:27.848491 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.848455    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvzs5\" (UniqueName: \"kubernetes.io/projected/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-api-access-tvzs5\") pod \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\" (UID: \"f06b92fd-2110-4e92-bcae-c0e6bb9ff08e\") "
Jul 28 16:35:27.848669 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.848646    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" (UID: "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:27.850490 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.850468    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" (UID: "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:35:27.850576 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.850512    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-api-access-tvzs5" (OuterVolumeSpecName: "kube-api-access-tvzs5") pod "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" (UID: "f06b92fd-2110-4e92-bcae-c0e6bb9ff08e"). InnerVolumeSpecName "kube-api-access-tvzs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:35:27.949770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.949736    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzs5\" (UniqueName: \"kubernetes.io/projected/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-api-access-tvzs5\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:27.949770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.949762    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:27.949770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:27.949772    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:28.128280 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:28.128251    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"]
Jul 28 16:35:28.131475 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:28.131452    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-9ed13-predictor-577847d7b6-4vgn2"]
Jul 28 16:35:28.811709 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:28.811669    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerStarted","Data":"0787e13b236cc016ca13d94c67c64cfdf0a573f653efda53ea2ba39a28c44d4a"}
Jul 28 16:35:28.811878 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:28.811720    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerStarted","Data":"a046ca11ec948c949699637cea8d8e144b837c664f7027691f60a0f84fe10e49"}
Jul 28 16:35:29.173026 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:29.172952    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" path="/var/lib/kubelet/pods/f06b92fd-2110-4e92-bcae-c0e6bb9ff08e/volumes"
Jul 28 16:35:30.454991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:30.454942    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 16:35:31.825409 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:31.825351    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerID="0787e13b236cc016ca13d94c67c64cfdf0a573f653efda53ea2ba39a28c44d4a" exitCode=0
Jul 28 16:35:31.825409 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:31.825404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerDied","Data":"0787e13b236cc016ca13d94c67c64cfdf0a573f653efda53ea2ba39a28c44d4a"}
Jul 28 16:35:32.830508 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.830469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerStarted","Data":"9756382a9e001079d2aba7fec3187dea51e5bbde1c6d384f49e15aed79d13966"}
Jul 28 16:35:32.830895 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.830537    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerStarted","Data":"133affa4d1e0524828ffc20538c4d95f5bb9794432cae76b711210de90373bdb"}
Jul 28 16:35:32.830895 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.830832    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:32.831018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.831001    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:32.832403 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.832365    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:35:32.832920 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.832902    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae69928a-1e66-435e-b05f-7ca27835572e" containerID="a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482" exitCode=0
Jul 28 16:35:32.832993 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.832975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerDied","Data":"a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482"}
Jul 28 16:35:32.851126 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:32.851082    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podStartSLOduration=2.269833165 podStartE2EDuration="5.851070339s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.581237174s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:35:32.850316604 +0000 UTC m=+918.243633198" watchObservedRunningTime="2026-07-28 16:35:32.851070339 +0000 UTC m=+918.244386878"
Jul 28 16:35:33.837595 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:33.837557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:35:34.841237 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:34.841202    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:35:35.455054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:35.455008    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 16:35:35.459502 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:35.459464    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:35:35.459782 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:35.459760    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:39.845367 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:39.845340    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:35:39.845857 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:39.845826    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:35:40.454885 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:40.454849    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 16:35:40.455090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:40.454995    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:45.454090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:45.454056    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 16:35:45.459590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:45.459565    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:35:45.459844 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:45.459818    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:49.846017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:49.845983    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:35:50.453920 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:50.453876    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 16:35:55.454596 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:55.454557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 16:35:55.460025 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:55.460003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 16:35:55.460137 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:55.460123    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:55.460394 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:55.460357    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:35:55.460474 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:55.460463    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:57.342820 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:57.342792    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae69928a_1e66_435e_b05f_7ca27835572e.slice/crio-1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 16:35:57.448922 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.448901    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:57.571290 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571225    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls\") pod \"ae69928a-1e66-435e-b05f-7ca27835572e\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") "
Jul 28 16:35:57.571455 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571297    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae69928a-1e66-435e-b05f-7ca27835572e-kserve-provision-location\") pod \"ae69928a-1e66-435e-b05f-7ca27835572e\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") "
Jul 28 16:35:57.571455 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571323    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-cabundle-cert\") pod \"ae69928a-1e66-435e-b05f-7ca27835572e\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") "
Jul 28 16:35:57.571455 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571343    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-kube-rbac-proxy-sar-config\") pod \"ae69928a-1e66-435e-b05f-7ca27835572e\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") "
Jul 28 16:35:57.571455 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571364    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wggbp\" (UniqueName: \"kubernetes.io/projected/ae69928a-1e66-435e-b05f-7ca27835572e-kube-api-access-wggbp\") pod \"ae69928a-1e66-435e-b05f-7ca27835572e\" (UID: \"ae69928a-1e66-435e-b05f-7ca27835572e\") "
Jul 28 16:35:57.571624 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571551    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae69928a-1e66-435e-b05f-7ca27835572e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae69928a-1e66-435e-b05f-7ca27835572e" (UID: "ae69928a-1e66-435e-b05f-7ca27835572e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:35:57.571661 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571641    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae69928a-1e66-435e-b05f-7ca27835572e-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:57.571707 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571668    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae69928a-1e66-435e-b05f-7ca27835572e" (UID: "ae69928a-1e66-435e-b05f-7ca27835572e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:57.571707 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.571676    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae69928a-1e66-435e-b05f-7ca27835572e" (UID: "ae69928a-1e66-435e-b05f-7ca27835572e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:35:57.573253 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.573234    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae69928a-1e66-435e-b05f-7ca27835572e" (UID: "ae69928a-1e66-435e-b05f-7ca27835572e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:35:57.573606 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.573579    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae69928a-1e66-435e-b05f-7ca27835572e-kube-api-access-wggbp" (OuterVolumeSpecName: "kube-api-access-wggbp") pod "ae69928a-1e66-435e-b05f-7ca27835572e" (UID: "ae69928a-1e66-435e-b05f-7ca27835572e"). InnerVolumeSpecName "kube-api-access-wggbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:35:57.672789 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.672766    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:57.672789 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.672787    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae69928a-1e66-435e-b05f-7ca27835572e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:57.672909 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.672796    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggbp\" (UniqueName: \"kubernetes.io/projected/ae69928a-1e66-435e-b05f-7ca27835572e-kube-api-access-wggbp\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:57.672909 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.672804    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae69928a-1e66-435e-b05f-7ca27835572e-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:35:57.931301 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.931223    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae69928a-1e66-435e-b05f-7ca27835572e" containerID="1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d" exitCode=0
Jul 28 16:35:57.931301 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.931261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerDied","Data":"1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d"}
Jul 28 16:35:57.931301 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.931277    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"
Jul 28 16:35:57.931556 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.931330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg" event={"ID":"ae69928a-1e66-435e-b05f-7ca27835572e","Type":"ContainerDied","Data":"0d21288b9a5e67361a44f91efccf48a63a02c9594baa444abbf7f785e9395e76"}
Jul 28 16:35:57.931556 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.931396    2573 scope.go:117] "RemoveContainer" containerID="1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d"
Jul 28 16:35:57.939720 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.939706    2573 scope.go:117] "RemoveContainer" containerID="fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc"
Jul 28 16:35:57.947044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.947020    2573 scope.go:117] "RemoveContainer" containerID="a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482"
Jul 28 16:35:57.952506 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.952485    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"]
Jul 28 16:35:57.956067 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.955512    2573 scope.go:117] "RemoveContainer" containerID="212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718"
Jul 28 16:35:57.956067 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.955926    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-9ed13-predictor-7c557d587-k8dsg"]
Jul 28 16:35:57.964324 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.964310    2573 scope.go:117] "RemoveContainer" containerID="1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d"
Jul 28 16:35:57.964584 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:57.964566    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d\": container with ID starting with 1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d not found: ID does not exist" containerID="1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d"
Jul 28 16:35:57.964630 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.964595    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d"} err="failed to get container status \"1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d\": rpc error: code = NotFound desc = could not find container \"1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d\": container with ID starting with 1a07d6025280c62c7297ba128b9dce117397ac40267eaf4684ee9b602cf1a04d not found: ID does not exist"
Jul 28 16:35:57.964630 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.964612    2573 scope.go:117] "RemoveContainer" containerID="fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc"
Jul 28 16:35:57.964815 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:57.964801    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc\": container with ID starting with fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc not found: ID does not exist" containerID="fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc"
Jul 28 16:35:57.964852 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.964820    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc"} err="failed to get container status \"fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc\": rpc error: code = NotFound desc = could not find container \"fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc\": container with ID starting with fa8cc8bdba3f3996e259ea1e52dda06ca9a4a9ceaac924d1cb9bfa6c9e51e1cc not found: ID does not exist"
Jul 28 16:35:57.964852 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.964833    2573 scope.go:117] "RemoveContainer" containerID="a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482"
Jul 28 16:35:57.965035 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:57.965021    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482\": container with ID starting with a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482 not found: ID does not exist" containerID="a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482"
Jul 28 16:35:57.965078 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.965037    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482"} err="failed to get container status \"a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482\": rpc error: code = NotFound desc = could not find container \"a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482\": container with ID starting with a46e9ac69310078076a7fc9b96dae8321660f50fcdf361e94f2d355655c42482 not found: ID does not exist"
Jul 28 16:35:57.965078 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.965048    2573 scope.go:117] "RemoveContainer" containerID="212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718"
Jul 28 16:35:57.965265 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:35:57.965250    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718\": container with ID starting with 212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718 not found: ID does not exist" containerID="212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718"
Jul 28 16:35:57.965306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:57.965269    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718"} err="failed to get container status \"212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718\": rpc error: code = NotFound desc = could not find container \"212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718\": container with ID starting with 212ef801c65a89bb3130788093984c3c5211db9201490092b84fa69eeb989718 not found: ID does not exist"
Jul 28 16:35:59.171944 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:59.171913    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" path="/var/lib/kubelet/pods/ae69928a-1e66-435e-b05f-7ca27835572e/volumes"
Jul 28 16:35:59.846101 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:35:59.846062    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:36:09.845804 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:36:09.845752    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:36:19.846127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:36:19.846088    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:36:29.845832 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:36:29.845790    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:36:39.845902 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:36:39.845825    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:36:49.846742 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:36:49.846699    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:36:59.845769 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:36:59.845731    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:37:01.168815 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:01.168771    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:37:11.168515 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:11.168479    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:37:21.168952 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:21.168912    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:37:31.168627 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:31.168583    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:37:41.169240 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:41.169202    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:37:51.172960 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:51.172933    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:37:57.562349 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:57.562319    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"]
Jul 28 16:37:57.562719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:57.562572    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" containerID="cri-o://133affa4d1e0524828ffc20538c4d95f5bb9794432cae76b711210de90373bdb" gracePeriod=30
Jul 28 16:37:57.562719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:57.562602    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kube-rbac-proxy" containerID="cri-o://9756382a9e001079d2aba7fec3187dea51e5bbde1c6d384f49e15aed79d13966" gracePeriod=30
Jul 28 16:37:58.366529 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:58.366500    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerID="9756382a9e001079d2aba7fec3187dea51e5bbde1c6d384f49e15aed79d13966" exitCode=2
Jul 28 16:37:58.366699 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:58.366579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerDied","Data":"9756382a9e001079d2aba7fec3187dea51e5bbde1c6d384f49e15aed79d13966"}
Jul 28 16:37:59.842257 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:37:59.842220    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 16:38:01.168460 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:01.168420    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 16:38:02.740948 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.740913    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"]
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741196    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741206    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741226    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kserve-container"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741231    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kserve-container"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741240    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="storage-initializer"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741245    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="storage-initializer"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741251    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741256    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741261    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741267    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741272    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kube-rbac-proxy"
Jul 28 16:38:02.741299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741276    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kube-rbac-proxy"
Jul 28 16:38:02.741740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741341    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kube-rbac-proxy"
Jul 28 16:38:02.741740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741349    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kserve-container"
Jul 28 16:38:02.741740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741355    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="agent"
Jul 28 16:38:02.741740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741359    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06b92fd-2110-4e92-bcae-c0e6bb9ff08e" containerName="kube-rbac-proxy"
Jul 28 16:38:02.741740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.741365    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae69928a-1e66-435e-b05f-7ca27835572e" containerName="kserve-container"
Jul 28 16:38:02.744509 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.744493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.746902 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.746882    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 16:38:02.747699 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.747679    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 16:38:02.758762 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.758739    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"]
Jul 28 16:38:02.762194 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.762178    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"]
Jul 28 16:38:02.762287 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.762264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.764669 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.764652    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 16:38:02.778440 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.778423    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"]
Jul 28 16:38:02.837238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.837357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837254    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432a333f-0f95-430a-8efa-2b1ce8be5ebc-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.837357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837272    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.837357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6vsq\" (UniqueName: \"kubernetes.io/projected/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-api-access-v6vsq\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.837357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837345    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77d92\" (UniqueName: \"kubernetes.io/projected/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-api-access-77d92\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.837522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.837522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.837522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.837522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.837522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.837501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.938680 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.938827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.938827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.938827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432a333f-0f95-430a-8efa-2b1ce8be5ebc-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.938827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.938827 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938799    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6vsq\" (UniqueName: \"kubernetes.io/projected/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-api-access-v6vsq\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.938827 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:38:02.938806    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 16:38:02.939127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77d92\" (UniqueName: \"kubernetes.io/projected/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-api-access-77d92\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.939127 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:38:02.938862    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls podName:d9d6bf13-47e1-4d3d-88e0-edfb35aa6675 nodeName:}" failed. No retries permitted until 2026-07-28 16:38:03.438842929 +0000 UTC m=+1068.832159452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls") pod "isvc-canary-create-predictor-cb85cf55c-sphmz" (UID: "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 16:38:02.939127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.939127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.939127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.938960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.939339 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.939160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.939529 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.939506    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.939601 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.939526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.939645 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.939603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.939683 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.939644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.939751 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.939735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.941324 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.941308    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432a333f-0f95-430a-8efa-2b1ce8be5ebc-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:02.947499 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.947481    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77d92\" (UniqueName: \"kubernetes.io/projected/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-api-access-77d92\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:02.947651 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:02.947633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6vsq\" (UniqueName: \"kubernetes.io/projected/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-api-access-v6vsq\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:03.070943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.070894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:03.203667 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.203647    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"]
Jul 28 16:38:03.205787 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:38:03.205759    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432a333f_0f95_430a_8efa_2b1ce8be5ebc.slice/crio-a5b8388e633ca1033309d15f51e88b3a1ed0dc0e0a46bbd1f620c472210981cc WatchSource:0}: Error finding container a5b8388e633ca1033309d15f51e88b3a1ed0dc0e0a46bbd1f620c472210981cc: Status 404 returned error can't find the container with id a5b8388e633ca1033309d15f51e88b3a1ed0dc0e0a46bbd1f620c472210981cc
Jul 28 16:38:03.385800 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.385724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerStarted","Data":"fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98"}
Jul 28 16:38:03.385800 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.385783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerStarted","Data":"a5b8388e633ca1033309d15f51e88b3a1ed0dc0e0a46bbd1f620c472210981cc"}
Jul 28 16:38:03.443279 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.443236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:03.445702 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.445680    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-sphmz\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:03.654585 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.654508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:03.785654 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:03.785593    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"]
Jul 28 16:38:03.787617 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:38:03.787584    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d6bf13_47e1_4d3d_88e0_edfb35aa6675.slice/crio-6a44b071a925ca4d323621a60d5faf5eda46cd559e1d5cda6a00969cec929c4c WatchSource:0}: Error finding container 6a44b071a925ca4d323621a60d5faf5eda46cd559e1d5cda6a00969cec929c4c: Status 404 returned error can't find the container with id 6a44b071a925ca4d323621a60d5faf5eda46cd559e1d5cda6a00969cec929c4c
Jul 28 16:38:04.392173 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:04.392132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerStarted","Data":"ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00"}
Jul 28 16:38:04.392323 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:04.392208    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerStarted","Data":"6a44b071a925ca4d323621a60d5faf5eda46cd559e1d5cda6a00969cec929c4c"}
Jul 28 16:38:04.841696 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:04.841651    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 16:38:06.401001 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.400976    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerID="133affa4d1e0524828ffc20538c4d95f5bb9794432cae76b711210de90373bdb" exitCode=0
Jul 28 16:38:06.401330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.401015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerDied","Data":"133affa4d1e0524828ffc20538c4d95f5bb9794432cae76b711210de90373bdb"}
Jul 28 16:38:06.401330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.401065    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n" event={"ID":"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3","Type":"ContainerDied","Data":"a046ca11ec948c949699637cea8d8e144b837c664f7027691f60a0f84fe10e49"}
Jul 28 16:38:06.401330 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.401098    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a046ca11ec948c949699637cea8d8e144b837c664f7027691f60a0f84fe10e49"
Jul 28 16:38:06.412894 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.412877    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:38:06.463636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.463615    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jvck\" (UniqueName: \"kubernetes.io/projected/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-api-access-5jvck\") pod \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") "
Jul 28 16:38:06.463749 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.463654    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-proxy-tls\") pod \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") "
Jul 28 16:38:06.463749 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.463684    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kserve-provision-location\") pod \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") "
Jul 28 16:38:06.463749 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.463704    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-rbac-proxy-sar-config\") pod \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") "
Jul 28 16:38:06.463902 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.463757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-cabundle-cert\") pod \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\" (UID: \"c2b9b87e-bfbd-48dc-8208-e927dd60e1d3\") "
Jul 28 16:38:06.463967 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.463945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" (UID: "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:38:06.464109 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.464079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" (UID: "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:38:06.464210 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.464128    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" (UID: "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:38:06.466332 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.466307    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-api-access-5jvck" (OuterVolumeSpecName: "kube-api-access-5jvck") pod "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" (UID: "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3"). InnerVolumeSpecName "kube-api-access-5jvck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:38:06.466332 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.466317    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" (UID: "c2b9b87e-bfbd-48dc-8208-e927dd60e1d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:38:06.564345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.564324    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:38:06.564345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.564344    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:38:06.564476 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.564353    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:38:06.564476 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.564362    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:38:06.564476 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:06.564386    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvck\" (UniqueName: \"kubernetes.io/projected/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3-kube-api-access-5jvck\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:38:07.406745 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:07.406676    2573 generic.go:356] "Generic (PLEG): container finished" podID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerID="fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98" exitCode=0
Jul 28 16:38:07.407068 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:07.406745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerDied","Data":"fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98"}
Jul 28 16:38:07.407068 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:07.406784    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"
Jul 28 16:38:07.429712 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:07.429655    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"]
Jul 28 16:38:07.431819 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:07.431798    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6d6fe-predictor-7f8c689f58-9n28n"]
Jul 28 16:38:08.411601 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.411568    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerID="ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00" exitCode=0
Jul 28 16:38:08.412006 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.411646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerDied","Data":"ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00"}
Jul 28 16:38:08.413877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.413859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerStarted","Data":"d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940"}
Jul 28 16:38:08.413955 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.413903    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerStarted","Data":"b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d"}
Jul 28 16:38:08.414243 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.414162    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:08.414243 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.414217    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:08.415571 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.415548    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:08.452180 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:08.452142    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podStartSLOduration=2.7403669 podStartE2EDuration="6.452129107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711762207s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:38:08.451855328 +0000 UTC m=+1073.845171867" watchObservedRunningTime="2026-07-28 16:38:08.452129107 +0000 UTC m=+1073.845445646"
Jul 28 16:38:09.172217 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:09.172184    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" path="/var/lib/kubelet/pods/c2b9b87e-bfbd-48dc-8208-e927dd60e1d3/volumes"
Jul 28 16:38:09.419822 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:09.419783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerStarted","Data":"528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315"}
Jul 28 16:38:09.420201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:09.419851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerStarted","Data":"38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c"}
Jul 28 16:38:09.420201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:09.420055    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:09.420201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:09.420091    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:09.439829 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:09.439742    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podStartSLOduration=3.723739262 podStartE2EDuration="7.439727918s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.715988656s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:38:09.438995935 +0000 UTC m=+1074.832312474" watchObservedRunningTime="2026-07-28 16:38:09.439727918 +0000 UTC m=+1074.833044458"
Jul 28 16:38:10.424257 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:10.424224    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:10.425580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:10.425555    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:38:11.427659 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:11.427623    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:38:14.425072 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:14.425028    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:38:14.425694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:14.425669    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:16.432465 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:16.432421    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:38:16.432980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:16.432949    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:38:24.425952 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:24.425912    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:26.432901 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:26.432865    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:38:34.426077 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:34.426044    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:36.433607 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:36.433567    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:38:44.425919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:44.425880    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:46.433732 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:46.433696    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:38:54.426224 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:54.426187    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:38:56.433836 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:38:56.433797    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:39:04.426161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:04.426120    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:39:06.433402 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:06.433353    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 16:39:14.425862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:14.425828    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 16:39:16.433638 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:16.433609    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:39:24.426518 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:24.426489    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:39:27.828237 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:27.828210    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"]
Jul 28 16:39:27.828679 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:27.828482    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container" containerID="cri-o://38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c" gracePeriod=30
Jul 28 16:39:27.828679 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:27.828499    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kube-rbac-proxy" containerID="cri-o://528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315" gracePeriod=30
Jul 28 16:39:27.837125 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:27.837102    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"]
Jul 28 16:39:27.837315 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:27.837283    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container" containerID="cri-o://b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d" gracePeriod=30
Jul 28 16:39:27.837423 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:27.837367    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kube-rbac-proxy" containerID="cri-o://d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940" gracePeriod=30
Jul 28 16:39:28.736031 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:28.736000    2573 generic.go:356] "Generic (PLEG): container finished" podID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerID="d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940" exitCode=2
Jul 28 16:39:28.736212 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:28.736088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerDied","Data":"d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940"}
Jul 28 16:39:28.737930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:28.737909    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerID="528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315" exitCode=2
Jul 28 16:39:28.738055 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:28.737935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerDied","Data":"528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315"}
Jul 28 16:39:29.421268 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:29.421225    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 28 16:39:31.428360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:31.428311    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 16:39:32.198161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.198129    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:39:32.200643 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.200628    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:39:32.309636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6vsq\" (UniqueName: \"kubernetes.io/projected/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-api-access-v6vsq\") pod \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") "
Jul 28 16:39:32.309636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309599    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-rbac-proxy-sar-config\") pod \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") "
Jul 28 16:39:32.309636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309626    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432a333f-0f95-430a-8efa-2b1ce8be5ebc-proxy-tls\") pod \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309650    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-cabundle-cert\") pod \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309672    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77d92\" (UniqueName: \"kubernetes.io/projected/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-api-access-77d92\") pod \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309703    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-rbac-proxy-sar-config\") pod \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309738    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kserve-provision-location\") pod \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-cabundle-cert\") pod \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\" (UID: \"432a333f-0f95-430a-8efa-2b1ce8be5ebc\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309785    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls\") pod \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") "
Jul 28 16:39:32.309882 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.309813    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kserve-provision-location\") pod \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\" (UID: \"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675\") "
Jul 28 16:39:32.310200 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.310006    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" (UID: "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:39:32.310200 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.310007    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "432a333f-0f95-430a-8efa-2b1ce8be5ebc" (UID: "432a333f-0f95-430a-8efa-2b1ce8be5ebc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:39:32.310333 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.310101    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" (UID: "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:39:32.310333 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.310257    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" (UID: "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:39:32.310446 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.310358    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "432a333f-0f95-430a-8efa-2b1ce8be5ebc" (UID: "432a333f-0f95-430a-8efa-2b1ce8be5ebc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:39:32.310596 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.310574    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "432a333f-0f95-430a-8efa-2b1ce8be5ebc" (UID: "432a333f-0f95-430a-8efa-2b1ce8be5ebc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:39:32.311907 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.311885    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-api-access-v6vsq" (OuterVolumeSpecName: "kube-api-access-v6vsq") pod "432a333f-0f95-430a-8efa-2b1ce8be5ebc" (UID: "432a333f-0f95-430a-8efa-2b1ce8be5ebc"). InnerVolumeSpecName "kube-api-access-v6vsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:39:32.312112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.312093    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-api-access-77d92" (OuterVolumeSpecName: "kube-api-access-77d92") pod "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" (UID: "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675"). InnerVolumeSpecName "kube-api-access-77d92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:39:32.312398 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.312368    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432a333f-0f95-430a-8efa-2b1ce8be5ebc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "432a333f-0f95-430a-8efa-2b1ce8be5ebc" (UID: "432a333f-0f95-430a-8efa-2b1ce8be5ebc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:39:32.312537 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.312515    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" (UID: "d9d6bf13-47e1-4d3d-88e0-edfb35aa6675"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:39:32.410997 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.410963    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6vsq\" (UniqueName: \"kubernetes.io/projected/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-api-access-v6vsq\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.410997 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.410992    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.410997 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411002    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432a333f-0f95-430a-8efa-2b1ce8be5ebc-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411010    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411018    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77d92\" (UniqueName: \"kubernetes.io/projected/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-api-access-77d92\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411026    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411033    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432a333f-0f95-430a-8efa-2b1ce8be5ebc-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411041    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432a333f-0f95-430a-8efa-2b1ce8be5ebc-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411049    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.411161 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.411056    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:39:32.754611 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.754584    2573 generic.go:356] "Generic (PLEG): container finished" podID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerID="b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d" exitCode=0
Jul 28 16:39:32.755009 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.754642    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"
Jul 28 16:39:32.755009 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.754672    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerDied","Data":"b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d"}
Jul 28 16:39:32.755009 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.754746    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9" event={"ID":"432a333f-0f95-430a-8efa-2b1ce8be5ebc","Type":"ContainerDied","Data":"a5b8388e633ca1033309d15f51e88b3a1ed0dc0e0a46bbd1f620c472210981cc"}
Jul 28 16:39:32.755009 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.754783    2573 scope.go:117] "RemoveContainer" containerID="d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940"
Jul 28 16:39:32.756687 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.756661    2573 generic.go:356] "Generic (PLEG): container finished" podID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerID="38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c" exitCode=0
Jul 28 16:39:32.756786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.756700    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerDied","Data":"38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c"}
Jul 28 16:39:32.756786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.756752    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz" event={"ID":"d9d6bf13-47e1-4d3d-88e0-edfb35aa6675","Type":"ContainerDied","Data":"6a44b071a925ca4d323621a60d5faf5eda46cd559e1d5cda6a00969cec929c4c"}
Jul 28 16:39:32.756786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.756779    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"
Jul 28 16:39:32.764092 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.764070    2573 scope.go:117] "RemoveContainer" containerID="b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d"
Jul 28 16:39:32.771305 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.771287    2573 scope.go:117] "RemoveContainer" containerID="fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98"
Jul 28 16:39:32.778442 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.778422    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"]
Jul 28 16:39:32.779748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.779731    2573 scope.go:117] "RemoveContainer" containerID="d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940"
Jul 28 16:39:32.779985 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:32.779968    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940\": container with ID starting with d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940 not found: ID does not exist" containerID="d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940"
Jul 28 16:39:32.780044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.779994    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940"} err="failed to get container status \"d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940\": rpc error: code = NotFound desc = could not find container \"d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940\": container with ID starting with d00672c62faeb2b79ab8a683b345139499ede2867a2a5f01e0d679f51e07a940 not found: ID does not exist"
Jul 28 16:39:32.780044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.780010    2573 scope.go:117] "RemoveContainer" containerID="b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d"
Jul 28 16:39:32.780260 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:32.780230    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d\": container with ID starting with b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d not found: ID does not exist" containerID="b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d"
Jul 28 16:39:32.780344 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.780286    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d"} err="failed to get container status \"b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d\": rpc error: code = NotFound desc = could not find container \"b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d\": container with ID starting with b1438b17fec1d3df3151bfa3ef7643b4745559c3d36f630b82332deed7b41e0d not found: ID does not exist"
Jul 28 16:39:32.780344 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.780300    2573 scope.go:117] "RemoveContainer" containerID="fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98"
Jul 28 16:39:32.781019 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:32.780864    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98\": container with ID starting with fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98 not found: ID does not exist" containerID="fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98"
Jul 28 16:39:32.781019 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.780892    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98"} err="failed to get container status \"fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98\": rpc error: code = NotFound desc = could not find container \"fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98\": container with ID starting with fe0f40c343b1bda932450069608eb45f1e07f3f43c685438070ea4717c501d98 not found: ID does not exist"
Jul 28 16:39:32.781019 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.780913    2573 scope.go:117] "RemoveContainer" containerID="528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315"
Jul 28 16:39:32.782462 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.782334    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xh5s9"]
Jul 28 16:39:32.790506 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.790492    2573 scope.go:117] "RemoveContainer" containerID="38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c"
Jul 28 16:39:32.794536 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.794513    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"]
Jul 28 16:39:32.797881 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.797848    2573 scope.go:117] "RemoveContainer" containerID="ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00"
Jul 28 16:39:32.799297 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.799279    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-sphmz"]
Jul 28 16:39:32.804853 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.804840    2573 scope.go:117] "RemoveContainer" containerID="528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315"
Jul 28 16:39:32.805096 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:32.805079    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315\": container with ID starting with 528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315 not found: ID does not exist" containerID="528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315"
Jul 28 16:39:32.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.805102    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315"} err="failed to get container status \"528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315\": rpc error: code = NotFound desc = could not find container \"528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315\": container with ID starting with 528e46dcb56448a54508a99fcdcdd43a78513420a066420e736fa2a10bd94315 not found: ID does not exist"
Jul 28 16:39:32.805154 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.805116    2573 scope.go:117] "RemoveContainer" containerID="38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c"
Jul 28 16:39:32.805318 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:32.805303    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c\": container with ID starting with 38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c not found: ID does not exist" containerID="38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c"
Jul 28 16:39:32.805385 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.805322    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c"} err="failed to get container status \"38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c\": rpc error: code = NotFound desc = could not find container \"38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c\": container with ID starting with 38ce62ffd12ea75e9dee7201ac0613cd14e9083007a7cb5d6c8b44d4724edb0c not found: ID does not exist"
Jul 28 16:39:32.805385 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.805334    2573 scope.go:117] "RemoveContainer" containerID="ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00"
Jul 28 16:39:32.805599 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:32.805584    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00\": container with ID starting with ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00 not found: ID does not exist" containerID="ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00"
Jul 28 16:39:32.805649 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:32.805602    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00"} err="failed to get container status \"ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00\": rpc error: code = NotFound desc = could not find container \"ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00\": container with ID starting with ac3a8304a07e2a291bf8f2900b987d125e7d1bfe4bf24be8c597ccf9b62fae00 not found: ID does not exist"
Jul 28 16:39:33.029871 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.029808    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"]
Jul 28 16:39:33.030080 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030069    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="storage-initializer"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030082    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="storage-initializer"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030099    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030104    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030112    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="storage-initializer"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030117    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="storage-initializer"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030122    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030127    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030132 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030133    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030138    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030150    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030156    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030163    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="storage-initializer"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030169    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="storage-initializer"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030178    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030182    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030188    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030193    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030231    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030239    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030244    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" containerName="kserve-container"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030250    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kube-rbac-proxy"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030255    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b9b87e-bfbd-48dc-8208-e927dd60e1d3" containerName="kserve-container"
Jul 28 16:39:33.030410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.030260    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" containerName="kserve-container"
Jul 28 16:39:33.034937 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.034913    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.036237 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.036221    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"]
Jul 28 16:39:33.037201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037169    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:39:33.037201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037187    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:39:33.037453 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037236    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:39:33.037453 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037323    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 16:39:33.037582 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037567    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:39:33.037664 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037572    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:39:33.037736 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.037573    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 16:39:33.039325 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.039308    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.041191 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.041174    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 16:39:33.045513 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.045495    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"]
Jul 28 16:39:33.061688 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.061659    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"]
Jul 28 16:39:33.115549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c29968a6-cf97-4425-bb03-79940e2af98d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.115676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115553    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.115676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.115676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.115676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.115676 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpnp\" (UniqueName: \"kubernetes.io/projected/c29968a6-cf97-4425-bb03-79940e2af98d-kube-api-access-lhpnp\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.115851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c29968a6-cf97-4425-bb03-79940e2af98d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.115851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.115851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxcv\" (UniqueName: \"kubernetes.io/projected/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-api-access-kkxcv\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.115851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.115791    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2be2d1b-5fae-40f3-820f-e809c73232e9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.173480 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.173455    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432a333f-0f95-430a-8efa-2b1ce8be5ebc" path="/var/lib/kubelet/pods/432a333f-0f95-430a-8efa-2b1ce8be5ebc/volumes"
Jul 28 16:39:33.173911 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.173897    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d6bf13-47e1-4d3d-88e0-edfb35aa6675" path="/var/lib/kubelet/pods/d9d6bf13-47e1-4d3d-88e0-edfb35aa6675/volumes"
Jul 28 16:39:33.216595 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c29968a6-cf97-4425-bb03-79940e2af98d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.216689 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.216689 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.216793 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.216845 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.216903 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpnp\" (UniqueName: \"kubernetes.io/projected/c29968a6-cf97-4425-bb03-79940e2af98d-kube-api-access-lhpnp\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.216958 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c29968a6-cf97-4425-bb03-79940e2af98d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.216958 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.217054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxcv\" (UniqueName: \"kubernetes.io/projected/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-api-access-kkxcv\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.217054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.216990    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2be2d1b-5fae-40f3-820f-e809c73232e9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.217200 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:33.217183    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 16:39:33.217331 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.217306    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.217422 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:39:33.217321    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls podName:b2be2d1b-5fae-40f3-820f-e809c73232e9 nodeName:}" failed. No retries permitted until 2026-07-28 16:39:33.717300791 +0000 UTC m=+1159.110617316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls") pod "isvc-canary-promote-v2-predictor-86766c4958-w6s7k" (UID: "b2be2d1b-5fae-40f3-820f-e809c73232e9") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 16:39:33.217422 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.217405    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c29968a6-cf97-4425-bb03-79940e2af98d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.217550 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.217487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.217550 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.217487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.217656 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.217616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2be2d1b-5fae-40f3-820f-e809c73232e9-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.217971 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.217950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.219538 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.219520    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c29968a6-cf97-4425-bb03-79940e2af98d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.228142 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.228120    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpnp\" (UniqueName: \"kubernetes.io/projected/c29968a6-cf97-4425-bb03-79940e2af98d-kube-api-access-lhpnp\") pod \"isvc-canary-promote-predictor-d9759ffd5-ppfrd\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.228351 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.228335    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxcv\" (UniqueName: \"kubernetes.io/projected/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-api-access-kkxcv\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.346872 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.346796    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:33.472860 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.472803    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"]
Jul 28 16:39:33.474973 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:39:33.474949    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29968a6_cf97_4425_bb03_79940e2af98d.slice/crio-08cedd504971c67814883e119dcd91073bab6123b5a7cc5604f38240fc6d9d6b WatchSource:0}: Error finding container 08cedd504971c67814883e119dcd91073bab6123b5a7cc5604f38240fc6d9d6b: Status 404 returned error can't find the container with id 08cedd504971c67814883e119dcd91073bab6123b5a7cc5604f38240fc6d9d6b
Jul 28 16:39:33.722673 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.722636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.725144 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.725118    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-w6s7k\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:33.764171 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.764138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerStarted","Data":"f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b"}
Jul 28 16:39:33.764605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.764198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerStarted","Data":"08cedd504971c67814883e119dcd91073bab6123b5a7cc5604f38240fc6d9d6b"}
Jul 28 16:39:33.953821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:33.953794    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:34.084051 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:34.084029    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"]
Jul 28 16:39:34.086364 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:39:34.086335    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2be2d1b_5fae_40f3_820f_e809c73232e9.slice/crio-427907c5a2747e3ca321d040b6f6ee73336d3c7c6703c048087bf7ddf4bfde30 WatchSource:0}: Error finding container 427907c5a2747e3ca321d040b6f6ee73336d3c7c6703c048087bf7ddf4bfde30: Status 404 returned error can't find the container with id 427907c5a2747e3ca321d040b6f6ee73336d3c7c6703c048087bf7ddf4bfde30
Jul 28 16:39:34.769241 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:34.769212    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerStarted","Data":"5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e"}
Jul 28 16:39:34.769708 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:34.769272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerStarted","Data":"427907c5a2747e3ca321d040b6f6ee73336d3c7c6703c048087bf7ddf4bfde30"}
Jul 28 16:39:37.782002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:37.781967    2573 generic.go:356] "Generic (PLEG): container finished" podID="c29968a6-cf97-4425-bb03-79940e2af98d" containerID="f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b" exitCode=0
Jul 28 16:39:37.782392 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:37.782042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerDied","Data":"f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b"}
Jul 28 16:39:38.788053 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.788014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerStarted","Data":"f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b"}
Jul 28 16:39:38.788554 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.788082    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerStarted","Data":"04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d"}
Jul 28 16:39:38.788554 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.788296    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:38.788554 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.788351    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:38.789473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.789453    2573 generic.go:356] "Generic (PLEG): container finished" podID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerID="5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e" exitCode=0
Jul 28 16:39:38.789583 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.789506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerDied","Data":"5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e"}
Jul 28 16:39:38.789930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.789907    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:39:38.809548 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:38.809505    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podStartSLOduration=2.19866161 podStartE2EDuration="5.809488096s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.610826486s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:39:38.808272388 +0000 UTC m=+1164.201588926" watchObservedRunningTime="2026-07-28 16:39:38.809488096 +0000 UTC m=+1164.202804635"
Jul 28 16:39:39.796721 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:39.796684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerStarted","Data":"d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567"}
Jul 28 16:39:39.797187 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:39.796743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerStarted","Data":"bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff"}
Jul 28 16:39:39.797252 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:39.797182    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:39:39.816779 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:39.816734    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podStartSLOduration=3.144564524 podStartE2EDuration="6.816720738s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672156214s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:39:39.815492914 +0000 UTC m=+1165.208809455" watchObservedRunningTime="2026-07-28 16:39:39.816720738 +0000 UTC m=+1165.210037276"
Jul 28 16:39:44.797410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.797362    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:44.798063 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.797906    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:44.799015 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.798989    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:39:44.801463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.801445    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:39:44.801975 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.801951    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:39:44.802769 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.802753    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:39:44.816359 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:44.816338    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:39:45.819913 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:45.819881    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:39:54.802392 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:54.802346    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:39:55.820156 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:39:55.820114    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:40:04.802047 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:04.802008    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:40:05.820821 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:05.820778    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:40:14.802719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:14.802682    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:40:15.078159 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:15.078073    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:40:15.078629 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:15.078609    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:40:15.820785 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:15.820747    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:40:24.802602 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:24.802557    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:40:25.820082 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:25.820043    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:40:34.801980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:34.801943    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:40:35.820486 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:35.820444    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:40:44.802048 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:44.802006    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:40:45.820786 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:45.820750    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:40:54.168846 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:54.168818    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:40:55.821272 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:40:55.821240    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:41:03.127417 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:03.127364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"]
Jul 28 16:41:03.127804 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:03.127593    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" containerID="cri-o://04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d" gracePeriod=30
Jul 28 16:41:03.127804 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:03.127634    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kube-rbac-proxy" containerID="cri-o://f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b" gracePeriod=30
Jul 28 16:41:04.123428 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:04.123394    2573 generic.go:356] "Generic (PLEG): container finished" podID="c29968a6-cf97-4425-bb03-79940e2af98d" containerID="f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b" exitCode=2
Jul 28 16:41:04.123577 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:04.123471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerDied","Data":"f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b"}
Jul 28 16:41:04.168386 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:04.168353    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 16:41:04.797469 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:04.797429    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 16:41:07.062928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.062905    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:41:07.137126 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.137094    2573 generic.go:356] "Generic (PLEG): container finished" podID="c29968a6-cf97-4425-bb03-79940e2af98d" containerID="04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d" exitCode=0
Jul 28 16:41:07.137274 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.137154    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"
Jul 28 16:41:07.137274 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.137173    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerDied","Data":"04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d"}
Jul 28 16:41:07.137274 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.137246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd" event={"ID":"c29968a6-cf97-4425-bb03-79940e2af98d","Type":"ContainerDied","Data":"08cedd504971c67814883e119dcd91073bab6123b5a7cc5604f38240fc6d9d6b"}
Jul 28 16:41:07.137471 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.137287    2573 scope.go:117] "RemoveContainer" containerID="f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b"
Jul 28 16:41:07.147563 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.147532    2573 scope.go:117] "RemoveContainer" containerID="04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d"
Jul 28 16:41:07.154752 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.154736    2573 scope.go:117] "RemoveContainer" containerID="f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b"
Jul 28 16:41:07.161662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.161649    2573 scope.go:117] "RemoveContainer" containerID="f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b"
Jul 28 16:41:07.161943 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:41:07.161927    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b\": container with ID starting with f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b not found: ID does not exist" containerID="f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b"
Jul 28 16:41:07.161991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.161951    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b"} err="failed to get container status \"f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b\": rpc error: code = NotFound desc = could not find container \"f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b\": container with ID starting with f4c7fc0da5c69ca955e53f120aa165d0809d3401b7bab7a85127d2edbb4bdf0b not found: ID does not exist"
Jul 28 16:41:07.161991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.161966    2573 scope.go:117] "RemoveContainer" containerID="04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d"
Jul 28 16:41:07.162188 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:41:07.162171    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d\": container with ID starting with 04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d not found: ID does not exist" containerID="04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d"
Jul 28 16:41:07.162258 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.162197    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d"} err="failed to get container status \"04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d\": rpc error: code = NotFound desc = could not find container \"04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d\": container with ID starting with 04d9dadd47aa0b946265f3cf84081ab9e627dd7a35188e558c5fc43a54954a5d not found: ID does not exist"
Jul 28 16:41:07.162258 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.162235    2573 scope.go:117] "RemoveContainer" containerID="f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b"
Jul 28 16:41:07.162479 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:41:07.162463    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b\": container with ID starting with f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b not found: ID does not exist" containerID="f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b"
Jul 28 16:41:07.162529 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.162486    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b"} err="failed to get container status \"f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b\": rpc error: code = NotFound desc = could not find container \"f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b\": container with ID starting with f160dfc9efedb4d61ed4b327e7b3768e1baae6fee72f86d376fb471ffa5bde4b not found: ID does not exist"
Jul 28 16:41:07.232810 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.232789    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c29968a6-cf97-4425-bb03-79940e2af98d-proxy-tls\") pod \"c29968a6-cf97-4425-bb03-79940e2af98d\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") "
Jul 28 16:41:07.232935 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.232822    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhpnp\" (UniqueName: \"kubernetes.io/projected/c29968a6-cf97-4425-bb03-79940e2af98d-kube-api-access-lhpnp\") pod \"c29968a6-cf97-4425-bb03-79940e2af98d\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") "
Jul 28 16:41:07.232935 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.232848    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c29968a6-cf97-4425-bb03-79940e2af98d-kserve-provision-location\") pod \"c29968a6-cf97-4425-bb03-79940e2af98d\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") "
Jul 28 16:41:07.233046 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.232970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-cabundle-cert\") pod \"c29968a6-cf97-4425-bb03-79940e2af98d\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") "
Jul 28 16:41:07.233117 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.233098    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-kube-rbac-proxy-sar-config\") pod \"c29968a6-cf97-4425-bb03-79940e2af98d\" (UID: \"c29968a6-cf97-4425-bb03-79940e2af98d\") "
Jul 28 16:41:07.233175 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.233156    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29968a6-cf97-4425-bb03-79940e2af98d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c29968a6-cf97-4425-bb03-79940e2af98d" (UID: "c29968a6-cf97-4425-bb03-79940e2af98d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:41:07.233361 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.233342    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c29968a6-cf97-4425-bb03-79940e2af98d-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:41:07.233493 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.233428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c29968a6-cf97-4425-bb03-79940e2af98d" (UID: "c29968a6-cf97-4425-bb03-79940e2af98d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:41:07.233574 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.233523    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c29968a6-cf97-4425-bb03-79940e2af98d" (UID: "c29968a6-cf97-4425-bb03-79940e2af98d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:41:07.235319 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.235294    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29968a6-cf97-4425-bb03-79940e2af98d-kube-api-access-lhpnp" (OuterVolumeSpecName: "kube-api-access-lhpnp") pod "c29968a6-cf97-4425-bb03-79940e2af98d" (UID: "c29968a6-cf97-4425-bb03-79940e2af98d"). InnerVolumeSpecName "kube-api-access-lhpnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:41:07.235482 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.235469    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29968a6-cf97-4425-bb03-79940e2af98d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c29968a6-cf97-4425-bb03-79940e2af98d" (UID: "c29968a6-cf97-4425-bb03-79940e2af98d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:41:07.333982 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.333953    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:41:07.333982 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.333982    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c29968a6-cf97-4425-bb03-79940e2af98d-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:41:07.333982 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.333992    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhpnp\" (UniqueName: \"kubernetes.io/projected/c29968a6-cf97-4425-bb03-79940e2af98d-kube-api-access-lhpnp\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:41:07.334138 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.334001    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c29968a6-cf97-4425-bb03-79940e2af98d-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:41:07.458538 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.458513    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"]
Jul 28 16:41:07.462709 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:07.462692    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-ppfrd"]
Jul 28 16:41:09.171930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:41:09.171895    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" path="/var/lib/kubelet/pods/c29968a6-cf97-4425-bb03-79940e2af98d/volumes"
Jul 28 16:42:15.146501 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:15.146473    2573 scope.go:117] "RemoveContainer" containerID="9756382a9e001079d2aba7fec3187dea51e5bbde1c6d384f49e15aed79d13966"
Jul 28 16:42:15.153944 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:15.153927    2573 scope.go:117] "RemoveContainer" containerID="0787e13b236cc016ca13d94c67c64cfdf0a573f653efda53ea2ba39a28c44d4a"
Jul 28 16:42:15.160792 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:15.160773    2573 scope.go:117] "RemoveContainer" containerID="133affa4d1e0524828ffc20538c4d95f5bb9794432cae76b711210de90373bdb"
Jul 28 16:42:43.532994 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:43.532921    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"]
Jul 28 16:42:43.533524 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:43.533143    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" containerID="cri-o://bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff" gracePeriod=30
Jul 28 16:42:43.533524 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:43.533195    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kube-rbac-proxy" containerID="cri-o://d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567" gracePeriod=30
Jul 28 16:42:44.513989 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:44.513956    2573 generic.go:356] "Generic (PLEG): container finished" podID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerID="d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567" exitCode=2
Jul 28 16:42:44.514163 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:44.514032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerDied","Data":"d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567"}
Jul 28 16:42:44.803281 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:44.803195    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 16:42:45.820398 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:45.820337    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:42:47.470060 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.470032    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:42:47.527867 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.527838    2573 generic.go:356] "Generic (PLEG): container finished" podID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerID="bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff" exitCode=0
Jul 28 16:42:47.528000 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.527891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerDied","Data":"bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff"}
Jul 28 16:42:47.528000 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.527938    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k" event={"ID":"b2be2d1b-5fae-40f3-820f-e809c73232e9","Type":"ContainerDied","Data":"427907c5a2747e3ca321d040b6f6ee73336d3c7c6703c048087bf7ddf4bfde30"}
Jul 28 16:42:47.528000 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.527896    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"
Jul 28 16:42:47.528163 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.527966    2573 scope.go:117] "RemoveContainer" containerID="d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567"
Jul 28 16:42:47.535678 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.535662    2573 scope.go:117] "RemoveContainer" containerID="bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff"
Jul 28 16:42:47.542468 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.542445    2573 scope.go:117] "RemoveContainer" containerID="5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e"
Jul 28 16:42:47.548796 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.548783    2573 scope.go:117] "RemoveContainer" containerID="d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567"
Jul 28 16:42:47.549033 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:42:47.549013    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567\": container with ID starting with d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567 not found: ID does not exist" containerID="d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567"
Jul 28 16:42:47.549090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.549048    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567"} err="failed to get container status \"d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567\": rpc error: code = NotFound desc = could not find container \"d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567\": container with ID starting with d3030e3a7a48ea724d83bfa399fba66b4735a90ff0273c2fc5eee1453cf4c567 not found: ID does not exist"
Jul 28 16:42:47.549090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.549070    2573 scope.go:117] "RemoveContainer" containerID="bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff"
Jul 28 16:42:47.549289 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:42:47.549275    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff\": container with ID starting with bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff not found: ID does not exist" containerID="bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff"
Jul 28 16:42:47.549335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.549293    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff"} err="failed to get container status \"bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff\": rpc error: code = NotFound desc = could not find container \"bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff\": container with ID starting with bb611ee1fb327bf2917db4f986c82af68502b4cd63252904c9325e396c00e6ff not found: ID does not exist"
Jul 28 16:42:47.549335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.549305    2573 scope.go:117] "RemoveContainer" containerID="5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e"
Jul 28 16:42:47.549527 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:42:47.549510    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e\": container with ID starting with 5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e not found: ID does not exist" containerID="5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e"
Jul 28 16:42:47.549572 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.549534    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e"} err="failed to get container status \"5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e\": rpc error: code = NotFound desc = could not find container \"5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e\": container with ID starting with 5911fb02faf6e5ff3ea9ec3ef9fd5919567c6f4581e26ace9196c1e9cc8b184e not found: ID does not exist"
Jul 28 16:42:47.555875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.555861    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkxcv\" (UniqueName: \"kubernetes.io/projected/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-api-access-kkxcv\") pod \"b2be2d1b-5fae-40f3-820f-e809c73232e9\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") "
Jul 28 16:42:47.555936 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.555900    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2be2d1b-5fae-40f3-820f-e809c73232e9-kserve-provision-location\") pod \"b2be2d1b-5fae-40f3-820f-e809c73232e9\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") "
Jul 28 16:42:47.555936 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.555929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls\") pod \"b2be2d1b-5fae-40f3-820f-e809c73232e9\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") "
Jul 28 16:42:47.556002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.555951    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-rbac-proxy-sar-config\") pod \"b2be2d1b-5fae-40f3-820f-e809c73232e9\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") "
Jul 28 16:42:47.556002 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.555977    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-cabundle-cert\") pod \"b2be2d1b-5fae-40f3-820f-e809c73232e9\" (UID: \"b2be2d1b-5fae-40f3-820f-e809c73232e9\") "
Jul 28 16:42:47.556216 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.556192    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2be2d1b-5fae-40f3-820f-e809c73232e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2be2d1b-5fae-40f3-820f-e809c73232e9" (UID: "b2be2d1b-5fae-40f3-820f-e809c73232e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:42:47.556339 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.556317    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2be2d1b-5fae-40f3-820f-e809c73232e9" (UID: "b2be2d1b-5fae-40f3-820f-e809c73232e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:47.556417 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.556332    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2be2d1b-5fae-40f3-820f-e809c73232e9" (UID: "b2be2d1b-5fae-40f3-820f-e809c73232e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:47.557796 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.557776    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2be2d1b-5fae-40f3-820f-e809c73232e9" (UID: "b2be2d1b-5fae-40f3-820f-e809c73232e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:42:47.557946 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.557929    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-api-access-kkxcv" (OuterVolumeSpecName: "kube-api-access-kkxcv") pod "b2be2d1b-5fae-40f3-820f-e809c73232e9" (UID: "b2be2d1b-5fae-40f3-820f-e809c73232e9"). InnerVolumeSpecName "kube-api-access-kkxcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:42:47.656515 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.656493    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2be2d1b-5fae-40f3-820f-e809c73232e9-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:42:47.656515 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.656513    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:42:47.656637 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.656527    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2be2d1b-5fae-40f3-820f-e809c73232e9-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:42:47.656637 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.656540    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkxcv\" (UniqueName: \"kubernetes.io/projected/b2be2d1b-5fae-40f3-820f-e809c73232e9-kube-api-access-kkxcv\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:42:47.656637 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.656552    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2be2d1b-5fae-40f3-820f-e809c73232e9-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:42:47.848592 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.848562    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"]
Jul 28 16:42:47.852895 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:47.852875    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-w6s7k"]
Jul 28 16:42:48.617867 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.617838    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"]
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618098    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618108    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618117    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="storage-initializer"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618122    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="storage-initializer"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618131    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="storage-initializer"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618137    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="storage-initializer"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618144    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kube-rbac-proxy"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618149    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kube-rbac-proxy"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618161    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kube-rbac-proxy"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618165    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kube-rbac-proxy"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618174    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618180    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618223    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kube-rbac-proxy"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618234    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c29968a6-cf97-4425-bb03-79940e2af98d" containerName="kserve-container"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618247    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kube-rbac-proxy"
Jul 28 16:42:48.618328 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.618255    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" containerName="kserve-container"
Jul 28 16:42:48.625876 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.625856    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"]
Jul 28 16:42:48.626013 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.625992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.628713 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.628690    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 16:42:48.628857 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.628840    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:42:48.629216 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.629198    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.629322 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.629286    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:42:48.629590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.629573    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:42:48.629671 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.629589    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:42:48.629671 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.629619    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:42:48.629671 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.629589    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 16:42:48.630887 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.630869    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 16:42:48.636235 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.636218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"]
Jul 28 16:42:48.641527 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.641506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"]
Jul 28 16:42:48.662169 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.662264 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.662264 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdpw\" (UniqueName: \"kubernetes.io/projected/3735a9d1-8d67-48c6-b033-629393b69be5-kube-api-access-hpdpw\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.662264 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.662410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3735a9d1-8d67-48c6-b033-629393b69be5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.662410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867cq\" (UniqueName: \"kubernetes.io/projected/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-api-access-867cq\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.662410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.662508 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.662508 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.662571 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.662502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.763449 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3735a9d1-8d67-48c6-b033-629393b69be5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.763540 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867cq\" (UniqueName: \"kubernetes.io/projected/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-api-access-867cq\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.763540 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.763540 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.763540 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.763721 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.763721 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.763721 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.763864 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:42:48.763722    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 16:42:48.763864 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:42:48.763800    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls podName:3735a9d1-8d67-48c6-b033-629393b69be5 nodeName:}" failed. No retries permitted until 2026-07-28 16:42:49.26377979 +0000 UTC m=+1354.657096309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" (UID: "3735a9d1-8d67-48c6-b033-629393b69be5") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 16:42:48.763864 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3735a9d1-8d67-48c6-b033-629393b69be5-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.764016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.764016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763724    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdpw\" (UniqueName: \"kubernetes.io/projected/3735a9d1-8d67-48c6-b033-629393b69be5-kube-api-access-hpdpw\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.764016 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.763926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.764256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.764228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.764314 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.764297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.764493 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.764472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.764552 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.764538    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.765985 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.765968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.772356 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.772337    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdpw\" (UniqueName: \"kubernetes.io/projected/3735a9d1-8d67-48c6-b033-629393b69be5-kube-api-access-hpdpw\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:48.772474 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.772456    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867cq\" (UniqueName: \"kubernetes.io/projected/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-api-access-867cq\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:48.937051 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:48.936992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:49.067803 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.067641    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"]
Jul 28 16:42:49.070222 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:42:49.070197    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a9f4c2_13cc_4297_80f3_6e8a89c5ab24.slice/crio-28d662709e30ee318c12d907347b815f01129aec2117663ce730b3bde7edaaf9 WatchSource:0}: Error finding container 28d662709e30ee318c12d907347b815f01129aec2117663ce730b3bde7edaaf9: Status 404 returned error can't find the container with id 28d662709e30ee318c12d907347b815f01129aec2117663ce730b3bde7edaaf9
Jul 28 16:42:49.172397 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.172358    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2be2d1b-5fae-40f3-820f-e809c73232e9" path="/var/lib/kubelet/pods/b2be2d1b-5fae-40f3-820f-e809c73232e9/volumes"
Jul 28 16:42:49.267867 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.267837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:49.270234 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.270209    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-7lq55\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:49.539969 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.539878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerStarted","Data":"98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931"}
Jul 28 16:42:49.539969 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.539958    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerStarted","Data":"28d662709e30ee318c12d907347b815f01129aec2117663ce730b3bde7edaaf9"}
Jul 28 16:42:49.545179 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.545154    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:49.672616 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:49.672589    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"]
Jul 28 16:42:49.674818 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:42:49.674797    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3735a9d1_8d67_48c6_b033_629393b69be5.slice/crio-e0ab8ac8dcb67e4e873697d031928ad41516c4bee40416406a58ac4133107c33 WatchSource:0}: Error finding container e0ab8ac8dcb67e4e873697d031928ad41516c4bee40416406a58ac4133107c33: Status 404 returned error can't find the container with id e0ab8ac8dcb67e4e873697d031928ad41516c4bee40416406a58ac4133107c33
Jul 28 16:42:50.544948 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:50.544913    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerStarted","Data":"5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562"}
Jul 28 16:42:50.545108 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:50.544973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerStarted","Data":"e0ab8ac8dcb67e4e873697d031928ad41516c4bee40416406a58ac4133107c33"}
Jul 28 16:42:53.557446 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:53.557413    2573 generic.go:356] "Generic (PLEG): container finished" podID="3735a9d1-8d67-48c6-b033-629393b69be5" containerID="5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562" exitCode=0
Jul 28 16:42:53.557874 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:53.557488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerDied","Data":"5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562"}
Jul 28 16:42:53.559003 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:53.558987    2573 generic.go:356] "Generic (PLEG): container finished" podID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerID="98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931" exitCode=0
Jul 28 16:42:53.559089 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:53.559057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerDied","Data":"98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931"}
Jul 28 16:42:54.565134 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.565096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerStarted","Data":"655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e"}
Jul 28 16:42:54.565618 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.565196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerStarted","Data":"9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365"}
Jul 28 16:42:54.565618 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.565429    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:54.567208 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.567187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerStarted","Data":"6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b"}
Jul 28 16:42:54.567308 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.567233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerStarted","Data":"27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4"}
Jul 28 16:42:54.567487 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.567471    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:54.567547 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.567511    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:42:54.568944 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.568920    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:42:54.585063 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.585027    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podStartSLOduration=3.152600636 podStartE2EDuration="6.585015603s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.432414967s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:42:54.584632105 +0000 UTC m=+1359.977948655" watchObservedRunningTime="2026-07-28 16:42:54.585015603 +0000 UTC m=+1359.978332141"
Jul 28 16:42:54.603265 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:54.603230    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podStartSLOduration=2.923449592 podStartE2EDuration="6.603218668s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.679769076s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:42:54.601969428 +0000 UTC m=+1359.995285966" watchObservedRunningTime="2026-07-28 16:42:54.603218668 +0000 UTC m=+1359.996535269"
Jul 28 16:42:55.571580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:55.571487    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:42:55.571580 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:55.571527    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:42:55.572930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:55.572911    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:42:56.575624 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:42:56.575586    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:43:00.575437 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:00.575404    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:43:00.576058 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:00.576031    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:43:01.580011 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:01.579979    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:43:01.580560 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:01.580536    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:43:10.576143 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:10.576102    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:43:11.581339 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:11.581296    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:43:20.576273 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:20.576239    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:43:21.581203 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:21.581159    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:43:30.576352 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:30.576315    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:43:31.581302 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:31.581266    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:43:40.576063 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:40.576025    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:43:41.580603 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:41.580571    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:43:50.576728 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:50.576686    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:43:51.581018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:43:51.580977    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:44:00.576585 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:00.576548    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:44:01.581830 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:01.581802    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:44:08.599460 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:08.599367    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"]
Jul 28 16:44:08.599842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:08.599610    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" containerID="cri-o://9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365" gracePeriod=30
Jul 28 16:44:08.599842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:08.599637    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kube-rbac-proxy" containerID="cri-o://655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e" gracePeriod=30
Jul 28 16:44:08.854267 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:08.854198    2573 generic.go:356] "Generic (PLEG): container finished" podID="3735a9d1-8d67-48c6-b033-629393b69be5" containerID="655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e" exitCode=2
Jul 28 16:44:08.854399 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:08.854262    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerDied","Data":"655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e"}
Jul 28 16:44:11.576365 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:11.576328    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 16:44:11.580597 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:11.580563    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 16:44:12.536740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.536717    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:44:12.581139 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581113    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-cabundle-cert\") pod \"3735a9d1-8d67-48c6-b033-629393b69be5\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") "
Jul 28 16:44:12.581496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581147    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls\") pod \"3735a9d1-8d67-48c6-b033-629393b69be5\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") "
Jul 28 16:44:12.581496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581170    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3735a9d1-8d67-48c6-b033-629393b69be5-kserve-provision-location\") pod \"3735a9d1-8d67-48c6-b033-629393b69be5\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") "
Jul 28 16:44:12.581496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581297    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpdpw\" (UniqueName: \"kubernetes.io/projected/3735a9d1-8d67-48c6-b033-629393b69be5-kube-api-access-hpdpw\") pod \"3735a9d1-8d67-48c6-b033-629393b69be5\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") "
Jul 28 16:44:12.581496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581348    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-kube-rbac-proxy-sar-config\") pod \"3735a9d1-8d67-48c6-b033-629393b69be5\" (UID: \"3735a9d1-8d67-48c6-b033-629393b69be5\") "
Jul 28 16:44:12.581496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581474    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3735a9d1-8d67-48c6-b033-629393b69be5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3735a9d1-8d67-48c6-b033-629393b69be5" (UID: "3735a9d1-8d67-48c6-b033-629393b69be5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:44:12.581748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581506    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3735a9d1-8d67-48c6-b033-629393b69be5" (UID: "3735a9d1-8d67-48c6-b033-629393b69be5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:12.581748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581656    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3735a9d1-8d67-48c6-b033-629393b69be5" (UID: "3735a9d1-8d67-48c6-b033-629393b69be5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:44:12.581748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581724    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:44:12.581860 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.581752    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3735a9d1-8d67-48c6-b033-629393b69be5-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:44:12.583270 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.583250    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3735a9d1-8d67-48c6-b033-629393b69be5" (UID: "3735a9d1-8d67-48c6-b033-629393b69be5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:44:12.583367 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.583343    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3735a9d1-8d67-48c6-b033-629393b69be5-kube-api-access-hpdpw" (OuterVolumeSpecName: "kube-api-access-hpdpw") pod "3735a9d1-8d67-48c6-b033-629393b69be5" (UID: "3735a9d1-8d67-48c6-b033-629393b69be5"). InnerVolumeSpecName "kube-api-access-hpdpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:44:12.682605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.682552    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3735a9d1-8d67-48c6-b033-629393b69be5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:44:12.682605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.682572    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3735a9d1-8d67-48c6-b033-629393b69be5-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:44:12.682605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.682581    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpdpw\" (UniqueName: \"kubernetes.io/projected/3735a9d1-8d67-48c6-b033-629393b69be5-kube-api-access-hpdpw\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:44:12.870913 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.870883    2573 generic.go:356] "Generic (PLEG): container finished" podID="3735a9d1-8d67-48c6-b033-629393b69be5" containerID="9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365" exitCode=0
Jul 28 16:44:12.871036 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.870936    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"
Jul 28 16:44:12.871036 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.870956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerDied","Data":"9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365"}
Jul 28 16:44:12.871106 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.871051    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55" event={"ID":"3735a9d1-8d67-48c6-b033-629393b69be5","Type":"ContainerDied","Data":"e0ab8ac8dcb67e4e873697d031928ad41516c4bee40416406a58ac4133107c33"}
Jul 28 16:44:12.871106 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.871083    2573 scope.go:117] "RemoveContainer" containerID="655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e"
Jul 28 16:44:12.879422 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.879408    2573 scope.go:117] "RemoveContainer" containerID="9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365"
Jul 28 16:44:12.886923 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.886909    2573 scope.go:117] "RemoveContainer" containerID="5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562"
Jul 28 16:44:12.891443 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.891422    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"]
Jul 28 16:44:12.894538 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.894520    2573 scope.go:117] "RemoveContainer" containerID="655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.895922    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-7lq55"]
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:44:12.896034    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e\": container with ID starting with 655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e not found: ID does not exist" containerID="655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.896059    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e"} err="failed to get container status \"655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e\": rpc error: code = NotFound desc = could not find container \"655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e\": container with ID starting with 655c71adf64605ca57bf5aee7f8022ba91383f4967981fb89f3a89facdd4870e not found: ID does not exist"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.896082    2573 scope.go:117] "RemoveContainer" containerID="9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:44:12.896388    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365\": container with ID starting with 9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365 not found: ID does not exist" containerID="9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.896417    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365"} err="failed to get container status \"9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365\": rpc error: code = NotFound desc = could not find container \"9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365\": container with ID starting with 9bc59b369ff3cfbe8f56ff99349f80af0101c3a0cc879f8213e2ff0fff9a2365 not found: ID does not exist"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.896435    2573 scope.go:117] "RemoveContainer" containerID="5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:44:12.896713    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562\": container with ID starting with 5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562 not found: ID does not exist" containerID="5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562"
Jul 28 16:44:12.897357 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:12.896744    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562"} err="failed to get container status \"5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562\": rpc error: code = NotFound desc = could not find container \"5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562\": container with ID starting with 5ac27da0ff4005de5aa4c94d886e05e34d0d21d338ada5830a7fd1e1daa06562 not found: ID does not exist"
Jul 28 16:44:13.172035 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:44:13.172003    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" path="/var/lib/kubelet/pods/3735a9d1-8d67-48c6-b033-629393b69be5/volumes"
Jul 28 16:45:15.101296 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:15.101262    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:45:15.103884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:15.103866    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:45:28.756515 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:28.756482    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"]
Jul 28 16:45:28.757087 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:28.756772    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" containerID="cri-o://27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4" gracePeriod=30
Jul 28 16:45:28.757087 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:28.756821    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kube-rbac-proxy" containerID="cri-o://6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b" gracePeriod=30
Jul 28 16:45:29.162105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:29.162031    2573 generic.go:356] "Generic (PLEG): container finished" podID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerID="6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b" exitCode=2
Jul 28 16:45:29.162235 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:29.162105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerDied","Data":"6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b"}
Jul 28 16:45:30.572442 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:30.572408    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 16:45:30.576745 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:30.576724    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:45:32.707853 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.707830    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:45:32.752256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752198    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-cabundle-cert\") pod \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") "
Jul 28 16:45:32.752256 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752233    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kserve-provision-location\") pod \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") "
Jul 28 16:45:32.752430 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752259    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-proxy-tls\") pod \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") "
Jul 28 16:45:32.752430 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752284    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-rbac-proxy-sar-config\") pod \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") "
Jul 28 16:45:32.752430 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752335    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-867cq\" (UniqueName: \"kubernetes.io/projected/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-api-access-867cq\") pod \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\" (UID: \"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24\") "
Jul 28 16:45:32.752603 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752575    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" (UID: "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:45:32.752673 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752598    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" (UID: "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:45:32.752673 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.752659    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" (UID: "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:45:32.754337 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.754316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" (UID: "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:45:32.754529 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.754508    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-api-access-867cq" (OuterVolumeSpecName: "kube-api-access-867cq") pod "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" (UID: "86a9f4c2-13cc-4297-80f3-6e8a89c5ab24"). InnerVolumeSpecName "kube-api-access-867cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:45:32.853236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.853216    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:45:32.853236 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.853234    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:45:32.853355 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.853244    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:45:32.853355 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.853253    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:45:32.853355 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:32.853260    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867cq\" (UniqueName: \"kubernetes.io/projected/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24-kube-api-access-867cq\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:45:33.180704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.180544    2573 generic.go:356] "Generic (PLEG): container finished" podID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerID="27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4" exitCode=0
Jul 28 16:45:33.180704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.180616    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerDied","Data":"27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4"}
Jul 28 16:45:33.180704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.180677    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74" event={"ID":"86a9f4c2-13cc-4297-80f3-6e8a89c5ab24","Type":"ContainerDied","Data":"28d662709e30ee318c12d907347b815f01129aec2117663ce730b3bde7edaaf9"}
Jul 28 16:45:33.180885 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.180718    2573 scope.go:117] "RemoveContainer" containerID="6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b"
Jul 28 16:45:33.180885 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.180728    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"
Jul 28 16:45:33.188548 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.188532    2573 scope.go:117] "RemoveContainer" containerID="27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4"
Jul 28 16:45:33.195424 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.195408    2573 scope.go:117] "RemoveContainer" containerID="98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931"
Jul 28 16:45:33.198963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.198942    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"]
Jul 28 16:45:33.202935 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.202895    2573 scope.go:117] "RemoveContainer" containerID="6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b"
Jul 28 16:45:33.203221 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:45:33.203197    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b\": container with ID starting with 6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b not found: ID does not exist" containerID="6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b"
Jul 28 16:45:33.203281 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.203230    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b"} err="failed to get container status \"6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b\": rpc error: code = NotFound desc = could not find container \"6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b\": container with ID starting with 6481b088e0f96d565cfcd77c62564a58f40562dc1366aeb3cff30c827b95bc5b not found: ID does not exist"
Jul 28 16:45:33.203281 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.203246    2573 scope.go:117] "RemoveContainer" containerID="27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4"
Jul 28 16:45:33.203517 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:45:33.203502    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4\": container with ID starting with 27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4 not found: ID does not exist" containerID="27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4"
Jul 28 16:45:33.203570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.203524    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4"} err="failed to get container status \"27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4\": rpc error: code = NotFound desc = could not find container \"27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4\": container with ID starting with 27b0fe730aabfaabc25239dbea5792314fb4635c74f8a2550b77af9b7f7490a4 not found: ID does not exist"
Jul 28 16:45:33.203570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.203537    2573 scope.go:117] "RemoveContainer" containerID="98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931"
Jul 28 16:45:33.203793 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:45:33.203770    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931\": container with ID starting with 98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931 not found: ID does not exist" containerID="98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931"
Jul 28 16:45:33.203837 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.203802    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931"} err="failed to get container status \"98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931\": rpc error: code = NotFound desc = could not find container \"98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931\": container with ID starting with 98355400f80a58dac1f73a7b7e4d3f285c8941bf8bee74c043206f4bc5c53931 not found: ID does not exist"
Jul 28 16:45:33.204122 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.204109    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-qvr74"]
Jul 28 16:45:33.989210 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989175    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"]
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989460    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kube-rbac-proxy"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989472    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kube-rbac-proxy"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989482    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989487    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989494    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989499    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989509    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kube-rbac-proxy"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989515    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kube-rbac-proxy"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989522    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="storage-initializer"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989528    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="storage-initializer"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989538    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="storage-initializer"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989543    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="storage-initializer"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989585    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kube-rbac-proxy"
Jul 28 16:45:33.989584 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989592    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kube-rbac-proxy"
Jul 28 16:45:33.989997 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989600    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3735a9d1-8d67-48c6-b033-629393b69be5" containerName="kserve-container"
Jul 28 16:45:33.989997 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.989606    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" containerName="kserve-container"
Jul 28 16:45:33.994223 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.994203    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:33.998253 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998233    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 16:45:33.998386 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998242    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:45:33.998532 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998514    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:45:33.998590 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998560    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 16:45:33.998636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998617    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:45:33.998920 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998905    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:45:33.998991 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:33.998974    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:45:34.011928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.011904    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"]
Jul 28 16:45:34.052701 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.052679    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"]
Jul 28 16:45:34.055914 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.055894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.058366 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.058346    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 16:45:34.061605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.061586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.061704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.061620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzfkk\" (UniqueName: \"kubernetes.io/projected/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-api-access-bzfkk\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.061704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.061651    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.061704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.061677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.061704 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.061701    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.073041 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.073021    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"]
Jul 28 16:45:34.162562 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.162770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzfkk\" (UniqueName: \"kubernetes.io/projected/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-api-access-bzfkk\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.162770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.162770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4nb\" (UniqueName: \"kubernetes.io/projected/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-api-access-bk4nb\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.162770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.162770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162717    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.162770 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.163076 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.163076 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942ab7e5-a937-49ad-b7ec-821f0e1eb850-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.163076 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.162882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.163076 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.163050    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.163475 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.163455    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.163552 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.163455    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.165426 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.165404    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.171606 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.171589    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzfkk\" (UniqueName: \"kubernetes.io/projected/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-api-access-bzfkk\") pod \"isvc-canary-stop-predictor-86dc9fdb65-7kw22\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.263669 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.263592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942ab7e5-a937-49ad-b7ec-821f0e1eb850-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.263669 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.263639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.263893 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.263767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.263893 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.263824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4nb\" (UniqueName: \"kubernetes.io/projected/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-api-access-bk4nb\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.263893 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.263885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.264101 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.264080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.264297 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.264278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.264510 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.264491    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.266164 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.266147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942ab7e5-a937-49ad-b7ec-821f0e1eb850-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.275499 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.275471    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4nb\" (UniqueName: \"kubernetes.io/projected/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-api-access-bk4nb\") pod \"isvc-canary-stop-v2-predictor-77f778d864-vjqjd\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.304059 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.304037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:34.366613 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.366524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:34.440873 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.440848    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"]
Jul 28 16:45:34.443140 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:45:34.443106    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7382f39_a4cd_4cac_bd06_74f83fa67a4f.slice/crio-3142a4329145ce6789d22e5eafb5c609387b529a83dc6e164f8b9b3e322eb4ec WatchSource:0}: Error finding container 3142a4329145ce6789d22e5eafb5c609387b529a83dc6e164f8b9b3e322eb4ec: Status 404 returned error can't find the container with id 3142a4329145ce6789d22e5eafb5c609387b529a83dc6e164f8b9b3e322eb4ec
Jul 28 16:45:34.506166 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:34.506143    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"]
Jul 28 16:45:34.510339 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:45:34.510314    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942ab7e5_a937_49ad_b7ec_821f0e1eb850.slice/crio-511ca77d50aebd34c9376323c1291511dc6c9aafb4e55518bcd243a245894240 WatchSource:0}: Error finding container 511ca77d50aebd34c9376323c1291511dc6c9aafb4e55518bcd243a245894240: Status 404 returned error can't find the container with id 511ca77d50aebd34c9376323c1291511dc6c9aafb4e55518bcd243a245894240
Jul 28 16:45:35.174240 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:35.174202    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a9f4c2-13cc-4297-80f3-6e8a89c5ab24" path="/var/lib/kubelet/pods/86a9f4c2-13cc-4297-80f3-6e8a89c5ab24/volumes"
Jul 28 16:45:35.191365 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:35.191334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerStarted","Data":"568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8"}
Jul 28 16:45:35.191544 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:35.191411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerStarted","Data":"511ca77d50aebd34c9376323c1291511dc6c9aafb4e55518bcd243a245894240"}
Jul 28 16:45:35.192812 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:35.192789    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerStarted","Data":"c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2"}
Jul 28 16:45:35.192906 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:35.192841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerStarted","Data":"3142a4329145ce6789d22e5eafb5c609387b529a83dc6e164f8b9b3e322eb4ec"}
Jul 28 16:45:38.204826 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:38.204791    2573 generic.go:356] "Generic (PLEG): container finished" podID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerID="568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8" exitCode=0
Jul 28 16:45:38.205214 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:38.204864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerDied","Data":"568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8"}
Jul 28 16:45:38.206360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:38.206341    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerID="c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2" exitCode=0
Jul 28 16:45:38.206466 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:38.206369    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerDied","Data":"c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2"}
Jul 28 16:45:39.212558 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.212516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerStarted","Data":"c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538"}
Jul 28 16:45:39.212980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.212584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerStarted","Data":"b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc"}
Jul 28 16:45:39.212980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.212875    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:39.213099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.213034    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:39.214526 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.214498    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:45:39.214699 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.214678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerStarted","Data":"f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b"}
Jul 28 16:45:39.214765 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.214751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerStarted","Data":"1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681"}
Jul 28 16:45:39.214945 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.214932    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:39.234965 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.234925    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podStartSLOduration=1.7837894300000001 podStartE2EDuration="5.234913065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.451123635s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:45:39.233159355 +0000 UTC m=+1524.626475893" watchObservedRunningTime="2026-07-28 16:45:39.234913065 +0000 UTC m=+1524.628229604"
Jul 28 16:45:39.255037 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:39.254991    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podStartSLOduration=2.711668951 podStartE2EDuration="6.254978155s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.543309204s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:45:39.252393399 +0000 UTC m=+1524.645709938" watchObservedRunningTime="2026-07-28 16:45:39.254978155 +0000 UTC m=+1524.648294693"
Jul 28 16:45:40.218737 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:40.218692    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:45:40.219102 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:40.218738    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:40.219841 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:40.219818    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:45:41.222140 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:41.222103    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:45:45.223025 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:45.222997    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:45:45.223480 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:45.223456    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:45:46.226838 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:46.226809    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:45:46.227306 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:46.227285    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:45:55.223859 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:55.223811    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:45:56.228051 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:45:56.228016    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:46:05.224114 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:05.224064    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:46:06.227494 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:06.227460    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:46:15.223619 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:15.223590    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:46:16.227661 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:16.227617    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:46:25.223424 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:25.223387    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:46:26.228104 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:26.228064    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:46:35.224282 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:35.224242    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:46:36.227990 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:36.227946    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:46:45.224433 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:45.224395    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:46:46.228517 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:46.228489    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:46:55.224133 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:55.224103    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:46:58.963388 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:58.963338    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"]
Jul 28 16:46:58.963865 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:58.963633    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container" containerID="cri-o://1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681" gracePeriod=30
Jul 28 16:46:58.963865 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:58.963704    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kube-rbac-proxy" containerID="cri-o://f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b" gracePeriod=30
Jul 28 16:46:58.998930 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:58.998906    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"]
Jul 28 16:46:58.999162 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:58.999140    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kube-rbac-proxy" containerID="cri-o://c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538" gracePeriod=30
Jul 28 16:46:58.999240 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:58.999139    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container" containerID="cri-o://b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc" gracePeriod=30
Jul 28 16:46:59.536655 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:59.536623    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerID="f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b" exitCode=2
Jul 28 16:46:59.536817 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:59.536696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerDied","Data":"f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b"}
Jul 28 16:46:59.538662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:59.538641    2573 generic.go:356] "Generic (PLEG): container finished" podID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerID="c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538" exitCode=2
Jul 28 16:46:59.538753 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:46:59.538684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerDied","Data":"c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538"}
Jul 28 16:47:00.219047 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:00.219010    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:47:01.223042 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:01.223003    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 16:47:03.018547 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.018523    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:47:03.038309 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.038281    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:47:03.129919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.129864    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942ab7e5-a937-49ad-b7ec-821f0e1eb850-proxy-tls\") pod \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") "
Jul 28 16:47:03.129919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.129896    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-proxy-tls\") pod \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") "
Jul 28 16:47:03.129919 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.129913    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-cabundle-cert\") pod \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") "
Jul 28 16:47:03.130152 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.129949    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kserve-provision-location\") pod \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") "
Jul 28 16:47:03.130152 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130009    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kserve-provision-location\") pod \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") "
Jul 28 16:47:03.130152 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-rbac-proxy-sar-config\") pod \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") "
Jul 28 16:47:03.130152 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130077    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-rbac-proxy-sar-config\") pod \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") "
Jul 28 16:47:03.130152 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130115    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-cabundle-cert\") pod \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") "
Jul 28 16:47:03.130419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130161    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk4nb\" (UniqueName: \"kubernetes.io/projected/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-api-access-bk4nb\") pod \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\" (UID: \"942ab7e5-a937-49ad-b7ec-821f0e1eb850\") "
Jul 28 16:47:03.130419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130192    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzfkk\" (UniqueName: \"kubernetes.io/projected/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-api-access-bzfkk\") pod \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\" (UID: \"e7382f39-a4cd-4cac-bd06-74f83fa67a4f\") "
Jul 28 16:47:03.130419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130211    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7382f39-a4cd-4cac-bd06-74f83fa67a4f" (UID: "e7382f39-a4cd-4cac-bd06-74f83fa67a4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:47:03.130419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130315    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "942ab7e5-a937-49ad-b7ec-821f0e1eb850" (UID: "942ab7e5-a937-49ad-b7ec-821f0e1eb850"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:47:03.130419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130324    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "942ab7e5-a937-49ad-b7ec-821f0e1eb850" (UID: "942ab7e5-a937-49ad-b7ec-821f0e1eb850"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:47:03.130662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130411    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7382f39-a4cd-4cac-bd06-74f83fa67a4f" (UID: "e7382f39-a4cd-4cac-bd06-74f83fa67a4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:47:03.130662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130537    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "942ab7e5-a937-49ad-b7ec-821f0e1eb850" (UID: "942ab7e5-a937-49ad-b7ec-821f0e1eb850"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:47:03.130662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130588    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7382f39-a4cd-4cac-bd06-74f83fa67a4f" (UID: "e7382f39-a4cd-4cac-bd06-74f83fa67a4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:47:03.130662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130593    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.130662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130614    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.130662 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.130627    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.132099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.132070    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942ab7e5-a937-49ad-b7ec-821f0e1eb850-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "942ab7e5-a937-49ad-b7ec-821f0e1eb850" (UID: "942ab7e5-a937-49ad-b7ec-821f0e1eb850"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:47:03.132363 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.132345    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-api-access-bzfkk" (OuterVolumeSpecName: "kube-api-access-bzfkk") pod "e7382f39-a4cd-4cac-bd06-74f83fa67a4f" (UID: "e7382f39-a4cd-4cac-bd06-74f83fa67a4f"). InnerVolumeSpecName "kube-api-access-bzfkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:47:03.132464 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.132364    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7382f39-a4cd-4cac-bd06-74f83fa67a4f" (UID: "e7382f39-a4cd-4cac-bd06-74f83fa67a4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:47:03.132505 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.132488    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-api-access-bk4nb" (OuterVolumeSpecName: "kube-api-access-bk4nb") pod "942ab7e5-a937-49ad-b7ec-821f0e1eb850" (UID: "942ab7e5-a937-49ad-b7ec-821f0e1eb850"). InnerVolumeSpecName "kube-api-access-bk4nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:47:03.231686 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.231659    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.231808 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.231716    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.232006 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.231983    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.232124 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.232109    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk4nb\" (UniqueName: \"kubernetes.io/projected/942ab7e5-a937-49ad-b7ec-821f0e1eb850-kube-api-access-bk4nb\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.232197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.232147    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzfkk\" (UniqueName: \"kubernetes.io/projected/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-kube-api-access-bzfkk\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.232197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.232163    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/942ab7e5-a937-49ad-b7ec-821f0e1eb850-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.232197 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.232176    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7382f39-a4cd-4cac-bd06-74f83fa67a4f-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:47:03.555440 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.555406    2573 generic.go:356] "Generic (PLEG): container finished" podID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerID="1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681" exitCode=0
Jul 28 16:47:03.555605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.555475    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"
Jul 28 16:47:03.555605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.555497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerDied","Data":"1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681"}
Jul 28 16:47:03.555605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.555558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22" event={"ID":"e7382f39-a4cd-4cac-bd06-74f83fa67a4f","Type":"ContainerDied","Data":"3142a4329145ce6789d22e5eafb5c609387b529a83dc6e164f8b9b3e322eb4ec"}
Jul 28 16:47:03.555605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.555585    2573 scope.go:117] "RemoveContainer" containerID="f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b"
Jul 28 16:47:03.557963 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.557937    2573 generic.go:356] "Generic (PLEG): container finished" podID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerID="b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc" exitCode=0
Jul 28 16:47:03.558090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.558007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerDied","Data":"b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc"}
Jul 28 16:47:03.558090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.558023    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"
Jul 28 16:47:03.558090 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.558046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd" event={"ID":"942ab7e5-a937-49ad-b7ec-821f0e1eb850","Type":"ContainerDied","Data":"511ca77d50aebd34c9376323c1291511dc6c9aafb4e55518bcd243a245894240"}
Jul 28 16:47:03.564087 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.564070    2573 scope.go:117] "RemoveContainer" containerID="1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681"
Jul 28 16:47:03.570852 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.570829    2573 scope.go:117] "RemoveContainer" containerID="c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2"
Jul 28 16:47:03.574702 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.574683    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"]
Jul 28 16:47:03.577819 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.577801    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-vjqjd"]
Jul 28 16:47:03.579303 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.579285    2573 scope.go:117] "RemoveContainer" containerID="f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b"
Jul 28 16:47:03.579561 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:03.579541    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b\": container with ID starting with f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b not found: ID does not exist" containerID="f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b"
Jul 28 16:47:03.579650 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.579570    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b"} err="failed to get container status \"f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b\": rpc error: code = NotFound desc = could not find container \"f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b\": container with ID starting with f0cf60dfae6b65b6ca54a94eb0a462784bd53f4fc7189226b5f77e56a0f8a18b not found: ID does not exist"
Jul 28 16:47:03.579650 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.579588    2573 scope.go:117] "RemoveContainer" containerID="1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681"
Jul 28 16:47:03.579840 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:03.579824    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681\": container with ID starting with 1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681 not found: ID does not exist" containerID="1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681"
Jul 28 16:47:03.579884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.579846    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681"} err="failed to get container status \"1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681\": rpc error: code = NotFound desc = could not find container \"1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681\": container with ID starting with 1e2b5fc9193c91ee809a43d25928875cf18e42121c012dbc9b30f5e589482681 not found: ID does not exist"
Jul 28 16:47:03.579884 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.579861    2573 scope.go:117] "RemoveContainer" containerID="c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2"
Jul 28 16:47:03.580082 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:03.580068    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2\": container with ID starting with c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2 not found: ID does not exist" containerID="c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2"
Jul 28 16:47:03.580127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.580086    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2"} err="failed to get container status \"c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2\": rpc error: code = NotFound desc = could not find container \"c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2\": container with ID starting with c7903ef13e54df3c903aaeb399268773ca84b4ac26fd3b08ec5dba76492350d2 not found: ID does not exist"
Jul 28 16:47:03.580127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.580098    2573 scope.go:117] "RemoveContainer" containerID="c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538"
Jul 28 16:47:03.586755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.586740    2573 scope.go:117] "RemoveContainer" containerID="b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc"
Jul 28 16:47:03.589505 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.589486    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"]
Jul 28 16:47:03.594105 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.593724    2573 scope.go:117] "RemoveContainer" containerID="568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8"
Jul 28 16:47:03.595473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.595411    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-7kw22"]
Jul 28 16:47:03.600869 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.600853    2573 scope.go:117] "RemoveContainer" containerID="c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538"
Jul 28 16:47:03.601119 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:03.601102    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538\": container with ID starting with c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538 not found: ID does not exist" containerID="c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538"
Jul 28 16:47:03.601171 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.601123    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538"} err="failed to get container status \"c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538\": rpc error: code = NotFound desc = could not find container \"c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538\": container with ID starting with c196cf79249b2491c5895a6ff501efd0afdd3eecf6ac56b314b09a235da3c538 not found: ID does not exist"
Jul 28 16:47:03.601171 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.601139    2573 scope.go:117] "RemoveContainer" containerID="b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc"
Jul 28 16:47:03.601359 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:03.601343    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc\": container with ID starting with b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc not found: ID does not exist" containerID="b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc"
Jul 28 16:47:03.601419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.601364    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc"} err="failed to get container status \"b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc\": rpc error: code = NotFound desc = could not find container \"b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc\": container with ID starting with b0d3a7a1406973887e1357f174d141ab365e9a2cb854d4bcb52b0fa84228d0dc not found: ID does not exist"
Jul 28 16:47:03.601419 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.601396    2573 scope.go:117] "RemoveContainer" containerID="568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8"
Jul 28 16:47:03.601605 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:03.601590    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8\": container with ID starting with 568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8 not found: ID does not exist" containerID="568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8"
Jul 28 16:47:03.601642 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:03.601609    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8"} err="failed to get container status \"568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8\": rpc error: code = NotFound desc = could not find container \"568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8\": container with ID starting with 568c5aecd419605e25e1def2728f3e9afe12b77d57b63afa84bc9f17e2f2d3d8 not found: ID does not exist"
Jul 28 16:47:05.172045 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:05.171978    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" path="/var/lib/kubelet/pods/942ab7e5-a937-49ad-b7ec-821f0e1eb850/volumes"
Jul 28 16:47:05.172438 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:05.172425    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" path="/var/lib/kubelet/pods/e7382f39-a4cd-4cac-bd06-74f83fa67a4f/volumes"
Jul 28 16:47:09.293101 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293067    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"]
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293329    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="storage-initializer"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293339    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="storage-initializer"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293351    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="storage-initializer"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293357    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="storage-initializer"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293392    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kube-rbac-proxy"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293403    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kube-rbac-proxy"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293418    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kube-rbac-proxy"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293425    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kube-rbac-proxy"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293438    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293445    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293455    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293460    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293507    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kube-rbac-proxy"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293513    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="942ab7e5-a937-49ad-b7ec-821f0e1eb850" containerName="kserve-container"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293519    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kube-rbac-proxy"
Jul 28 16:47:09.293549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.293526    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7382f39-a4cd-4cac-bd06-74f83fa67a4f" containerName="kserve-container"
Jul 28 16:47:09.298160 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.298144    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.302347 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.302323    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:47:09.302474 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.302364    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2a8c3a-kube-rbac-proxy-sar-config\""
Jul 28 16:47:09.302474 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.302366    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:47:09.303356 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.303339    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:47:09.303734 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.303719    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:47:09.303734 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.303728    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2a8c3a-predictor-serving-cert\""
Jul 28 16:47:09.303855 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.303838    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:47:09.308522 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.308505    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"]
Jul 28 16:47:09.373703 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.373683    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsflz\" (UniqueName: \"kubernetes.io/projected/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-api-access-lsflz\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.373816 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.373717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-cabundle-cert\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.373816 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.373745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kserve-provision-location\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.373894 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.373826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.373894 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.373860    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.474737 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.474716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kserve-provision-location\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.474817 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.474751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.474817 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.474777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.474817 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.474798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsflz\" (UniqueName: \"kubernetes.io/projected/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-api-access-lsflz\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.474966 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.474821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-cabundle-cert\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.474966 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:09.474877    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-serving-cert: secret "isvc-primary-2a8c3a-predictor-serving-cert" not found
Jul 28 16:47:09.474966 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:47:09.474946    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls podName:1e94e79f-271c-42f2-a910-eb4f95df0ba3 nodeName:}" failed. No retries permitted until 2026-07-28 16:47:09.974926503 +0000 UTC m=+1615.368243022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls") pod "isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" (UID: "1e94e79f-271c-42f2-a910-eb4f95df0ba3") : secret "isvc-primary-2a8c3a-predictor-serving-cert" not found
Jul 28 16:47:09.475119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.475085    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kserve-provision-location\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.475335 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.475317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-cabundle-cert\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.475458 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.475441    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.483358 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.483338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsflz\" (UniqueName: \"kubernetes.io/projected/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-api-access-lsflz\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.979535 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.979508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:09.981862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:09.981837    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls\") pod \"isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") " pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:10.207776 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:10.207746    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:10.356044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:10.356021    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"]
Jul 28 16:47:10.358874 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:47:10.358845    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e94e79f_271c_42f2_a910_eb4f95df0ba3.slice/crio-a164005b44a51998495bd5bc00a390d0abde3067be79fdfdc240d7a42585a324 WatchSource:0}: Error finding container a164005b44a51998495bd5bc00a390d0abde3067be79fdfdc240d7a42585a324: Status 404 returned error can't find the container with id a164005b44a51998495bd5bc00a390d0abde3067be79fdfdc240d7a42585a324
Jul 28 16:47:10.592979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:10.592902    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerStarted","Data":"9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39"}
Jul 28 16:47:10.592979 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:10.592960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerStarted","Data":"a164005b44a51998495bd5bc00a390d0abde3067be79fdfdc240d7a42585a324"}
Jul 28 16:47:14.608129 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:14.608094    2573 generic.go:356] "Generic (PLEG): container finished" podID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerID="9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39" exitCode=0
Jul 28 16:47:14.608511 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:14.608171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerDied","Data":"9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39"}
Jul 28 16:47:15.613830 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:15.613796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerStarted","Data":"6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884"}
Jul 28 16:47:15.614253 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:15.613877    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerStarted","Data":"a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef"}
Jul 28 16:47:15.614253 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:15.614124    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:15.614362 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:15.614267    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:15.615581 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:15.615558    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:47:15.635937 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:15.635896    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podStartSLOduration=3.102778905 podStartE2EDuration="6.635883823s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.533104918s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:47:15.634536223 +0000 UTC m=+1621.027852763" watchObservedRunningTime="2026-07-28 16:47:15.635883823 +0000 UTC m=+1621.029200361"
Jul 28 16:47:16.617487 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:16.617445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:47:17.620944 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:17.620905    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:47:22.625565 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:22.625535    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:47:22.626014 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:22.625988    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:47:32.626854 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:32.626810    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:47:42.626665 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:42.626624    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:47:52.626807 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:47:52.626764    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:48:02.626739 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:02.626693    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:48:12.626030 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:12.625991    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 16:48:22.627339 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:22.627312    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:48:44.545421 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.545329    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"]
Jul 28 16:48:44.548624 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.548607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.550795 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.550749    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-2a8c3a\""
Jul 28 16:48:44.550795 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.550749    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2a8c3a-predictor-serving-cert\""
Jul 28 16:48:44.550980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.550842    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-2a8c3a-dockercfg-q8tqd\""
Jul 28 16:48:44.550980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.550891    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2a8c3a-kube-rbac-proxy-sar-config\""
Jul 28 16:48:44.560059 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.560038    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"]
Jul 28 16:48:44.650871 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.650841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kserve-provision-location\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.651005 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.650874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.651005 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.650900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df8e4d27-5261-4b7b-be82-9a84d5065c6b-proxy-tls\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.651005 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.650993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-cabundle-cert\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.651121 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.651020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrhz\" (UniqueName: \"kubernetes.io/projected/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-api-access-dvrhz\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.751695 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.751662    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrhz\" (UniqueName: \"kubernetes.io/projected/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-api-access-dvrhz\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.751842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.751725    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kserve-provision-location\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.751842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.751746    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.751842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.751768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df8e4d27-5261-4b7b-be82-9a84d5065c6b-proxy-tls\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.751842 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.751795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-cabundle-cert\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.752233 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.752210    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kserve-provision-location\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.752495 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.752478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-cabundle-cert\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.752495 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.752487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.754137 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.754122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df8e4d27-5261-4b7b-be82-9a84d5065c6b-proxy-tls\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.760719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.760696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrhz\" (UniqueName: \"kubernetes.io/projected/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-api-access-dvrhz\") pod \"isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") " pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.858589 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.858532    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:48:44.989743 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:44.989719    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"]
Jul 28 16:48:44.992322 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:48:44.992292    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8e4d27_5261_4b7b_be82_9a84d5065c6b.slice/crio-a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411 WatchSource:0}: Error finding container a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411: Status 404 returned error can't find the container with id a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411
Jul 28 16:48:45.957162 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:45.957128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" event={"ID":"df8e4d27-5261-4b7b-be82-9a84d5065c6b","Type":"ContainerStarted","Data":"63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1"}
Jul 28 16:48:45.957553 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:45.957184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" event={"ID":"df8e4d27-5261-4b7b-be82-9a84d5065c6b","Type":"ContainerStarted","Data":"a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411"}
Jul 28 16:48:51.982644 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:51.982613    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/0.log"
Jul 28 16:48:51.982987 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:51.982662    2573 generic.go:356] "Generic (PLEG): container finished" podID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerID="63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1" exitCode=1
Jul 28 16:48:51.982987 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:51.982732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" event={"ID":"df8e4d27-5261-4b7b-be82-9a84d5065c6b","Type":"ContainerDied","Data":"63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1"}
Jul 28 16:48:52.988139 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:52.988112    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/0.log"
Jul 28 16:48:52.988549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:52.988204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" event={"ID":"df8e4d27-5261-4b7b-be82-9a84d5065c6b","Type":"ContainerStarted","Data":"e03d52f3169cd1584930853e61eb1f456fae1bd5a5619cd220262e5a023523c3"}
Jul 28 16:48:54.997429 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:54.997404    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/1.log"
Jul 28 16:48:54.997848 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:54.997779    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/0.log"
Jul 28 16:48:54.997848 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:54.997807    2573 generic.go:356] "Generic (PLEG): container finished" podID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerID="e03d52f3169cd1584930853e61eb1f456fae1bd5a5619cd220262e5a023523c3" exitCode=1
Jul 28 16:48:54.997928 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:54.997886    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" event={"ID":"df8e4d27-5261-4b7b-be82-9a84d5065c6b","Type":"ContainerDied","Data":"e03d52f3169cd1584930853e61eb1f456fae1bd5a5619cd220262e5a023523c3"}
Jul 28 16:48:54.997962 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:54.997950    2573 scope.go:117] "RemoveContainer" containerID="63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1"
Jul 28 16:48:54.998258 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:54.998242    2573 scope.go:117] "RemoveContainer" containerID="63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1"
Jul 28 16:48:55.008289 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:48:55.008267    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_kserve-ci-e2e-test(df8e4d27-5261-4b7b-be82-9a84d5065c6b)\"" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b"
Jul 28 16:48:55.008549 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:48:55.008529    2573 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_kserve-ci-e2e-test_df8e4d27-5261-4b7b-be82-9a84d5065c6b_0 in pod sandbox a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411 from index: no such id: '63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1'" containerID="63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1"
Jul 28 16:48:55.008603 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:55.008557    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_kserve-ci-e2e-test_df8e4d27-5261-4b7b-be82-9a84d5065c6b_0 in pod sandbox a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411 from index: no such id: '63cf49684979ac1790a4bda8cf0eba17800b5467807a5b7a7185a6536981dbd1'"
Jul 28 16:48:56.003388 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:48:56.003357    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/1.log"
Jul 28 16:49:02.616937 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.616906    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"]
Jul 28 16:49:02.688853 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.688821    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"]
Jul 28 16:49:02.689133 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.689110    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container" containerID="cri-o://a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef" gracePeriod=30
Jul 28 16:49:02.689242 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.689195    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kube-rbac-proxy" containerID="cri-o://6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884" gracePeriod=30
Jul 28 16:49:02.808283 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.808263    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/1.log"
Jul 28 16:49:02.810269 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.810253    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:49:02.900017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.899952    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-cabundle-cert\") pod \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") "
Jul 28 16:49:02.900017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.899981    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvrhz\" (UniqueName: \"kubernetes.io/projected/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-api-access-dvrhz\") pod \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") "
Jul 28 16:49:02.900017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.900016    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kserve-provision-location\") pod \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") "
Jul 28 16:49:02.900238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.900064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-rbac-proxy-sar-config\") pod \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") "
Jul 28 16:49:02.900238 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.900086    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df8e4d27-5261-4b7b-be82-9a84d5065c6b-proxy-tls\") pod \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\" (UID: \"df8e4d27-5261-4b7b-be82-9a84d5065c6b\") "
Jul 28 16:49:02.900345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.900305    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df8e4d27-5261-4b7b-be82-9a84d5065c6b" (UID: "df8e4d27-5261-4b7b-be82-9a84d5065c6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:49:02.900433 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.900365    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df8e4d27-5261-4b7b-be82-9a84d5065c6b" (UID: "df8e4d27-5261-4b7b-be82-9a84d5065c6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:49:02.900488 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.900467    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df8e4d27-5261-4b7b-be82-9a84d5065c6b" (UID: "df8e4d27-5261-4b7b-be82-9a84d5065c6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:49:02.902099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.902077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-api-access-dvrhz" (OuterVolumeSpecName: "kube-api-access-dvrhz") pod "df8e4d27-5261-4b7b-be82-9a84d5065c6b" (UID: "df8e4d27-5261-4b7b-be82-9a84d5065c6b"). InnerVolumeSpecName "kube-api-access-dvrhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:49:02.902237 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:02.902220    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8e4d27-5261-4b7b-be82-9a84d5065c6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df8e4d27-5261-4b7b-be82-9a84d5065c6b" (UID: "df8e4d27-5261-4b7b-be82-9a84d5065c6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:49:03.000679 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.000646    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:03.000679 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.000676    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrhz\" (UniqueName: \"kubernetes.io/projected/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-api-access-dvrhz\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:03.000820 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.000698    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:03.000820 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.000707    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df8e4d27-5261-4b7b-be82-9a84d5065c6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:03.000820 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.000715    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df8e4d27-5261-4b7b-be82-9a84d5065c6b-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:03.032912 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.032888    2573 generic.go:356] "Generic (PLEG): container finished" podID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerID="6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884" exitCode=2
Jul 28 16:49:03.033007 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.032959    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerDied","Data":"6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884"}
Jul 28 16:49:03.034174 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.034162    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc_df8e4d27-5261-4b7b-be82-9a84d5065c6b/storage-initializer/1.log"
Jul 28 16:49:03.034235 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.034208    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc" event={"ID":"df8e4d27-5261-4b7b-be82-9a84d5065c6b","Type":"ContainerDied","Data":"a57afeecfc82272e9d5edf5c38d1184291b01c6c0f8f0ec8e5cedfe173a7e411"}
Jul 28 16:49:03.034272 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.034239    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"
Jul 28 16:49:03.034272 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.034248    2573 scope.go:117] "RemoveContainer" containerID="e03d52f3169cd1584930853e61eb1f456fae1bd5a5619cd220262e5a023523c3"
Jul 28 16:49:03.077181 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.077161    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"]
Jul 28 16:49:03.081780 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.081760    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2a8c3a-predictor-5bc45d97d6-9zqnc"]
Jul 28 16:49:03.173592 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:03.173527    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" path="/var/lib/kubelet/pods/df8e4d27-5261-4b7b-be82-9a84d5065c6b/volumes"
Jul 28 16:49:06.725141 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.725120    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:49:06.831839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.831767    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-cabundle-cert\") pod \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") "
Jul 28 16:49:06.831839 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.831809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls\") pod \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") "
Jul 28 16:49:06.832017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.831856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-rbac-proxy-sar-config\") pod \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") "
Jul 28 16:49:06.832017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.831915    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsflz\" (UniqueName: \"kubernetes.io/projected/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-api-access-lsflz\") pod \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") "
Jul 28 16:49:06.832017 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.831957    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kserve-provision-location\") pod \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\" (UID: \"1e94e79f-271c-42f2-a910-eb4f95df0ba3\") "
Jul 28 16:49:06.832165 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.832122    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e94e79f-271c-42f2-a910-eb4f95df0ba3" (UID: "1e94e79f-271c-42f2-a910-eb4f95df0ba3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:49:06.832226 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.832203    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e94e79f-271c-42f2-a910-eb4f95df0ba3" (UID: "1e94e79f-271c-42f2-a910-eb4f95df0ba3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:49:06.832355 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.832333    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e94e79f-271c-42f2-a910-eb4f95df0ba3" (UID: "1e94e79f-271c-42f2-a910-eb4f95df0ba3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:49:06.833943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.833924    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e94e79f-271c-42f2-a910-eb4f95df0ba3" (UID: "1e94e79f-271c-42f2-a910-eb4f95df0ba3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:49:06.834121 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.834098    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-api-access-lsflz" (OuterVolumeSpecName: "kube-api-access-lsflz") pod "1e94e79f-271c-42f2-a910-eb4f95df0ba3" (UID: "1e94e79f-271c-42f2-a910-eb4f95df0ba3"). InnerVolumeSpecName "kube-api-access-lsflz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:49:06.932977 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.932953    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsflz\" (UniqueName: \"kubernetes.io/projected/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-api-access-lsflz\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:06.932977 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.932973    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:06.933104 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.932982    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:06.933104 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.932991    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e94e79f-271c-42f2-a910-eb4f95df0ba3-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:06.933104 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:06.932999    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e94e79f-271c-42f2-a910-eb4f95df0ba3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:07.053120 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.053088    2573 generic.go:356] "Generic (PLEG): container finished" podID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerID="a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef" exitCode=0
Jul 28 16:49:07.053229 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.053139    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"
Jul 28 16:49:07.053229 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.053153    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerDied","Data":"a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef"}
Jul 28 16:49:07.053229 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.053221    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5" event={"ID":"1e94e79f-271c-42f2-a910-eb4f95df0ba3","Type":"ContainerDied","Data":"a164005b44a51998495bd5bc00a390d0abde3067be79fdfdc240d7a42585a324"}
Jul 28 16:49:07.053354 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.053260    2573 scope.go:117] "RemoveContainer" containerID="6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884"
Jul 28 16:49:07.061851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.061834    2573 scope.go:117] "RemoveContainer" containerID="a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef"
Jul 28 16:49:07.074168 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.074092    2573 scope.go:117] "RemoveContainer" containerID="9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39"
Jul 28 16:49:07.075757 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.075736    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"]
Jul 28 16:49:07.081279 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.081260    2573 scope.go:117] "RemoveContainer" containerID="6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884"
Jul 28 16:49:07.081501 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.081482    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2a8c3a-predictor-55d8dc79df-zfbc5"]
Jul 28 16:49:07.081606 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:49:07.081587    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884\": container with ID starting with 6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884 not found: ID does not exist" containerID="6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884"
Jul 28 16:49:07.081646 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.081615    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884"} err="failed to get container status \"6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884\": rpc error: code = NotFound desc = could not find container \"6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884\": container with ID starting with 6eb5e72a14b6ec8fa86abf818b982536d492f5b50c8f01b72493a05d87098884 not found: ID does not exist"
Jul 28 16:49:07.081646 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.081631    2573 scope.go:117] "RemoveContainer" containerID="a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef"
Jul 28 16:49:07.081911 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:49:07.081842    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef\": container with ID starting with a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef not found: ID does not exist" containerID="a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef"
Jul 28 16:49:07.081911 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.081895    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef"} err="failed to get container status \"a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef\": rpc error: code = NotFound desc = could not find container \"a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef\": container with ID starting with a8fcb6d6d448b0673ed3aa990a3da499840e7bb3fe4b7857bb78b125909d4eef not found: ID does not exist"
Jul 28 16:49:07.081911 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.081910    2573 scope.go:117] "RemoveContainer" containerID="9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39"
Jul 28 16:49:07.082155 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:49:07.082139    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39\": container with ID starting with 9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39 not found: ID does not exist" containerID="9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39"
Jul 28 16:49:07.082201 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.082160    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39"} err="failed to get container status \"9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39\": rpc error: code = NotFound desc = could not find container \"9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39\": container with ID starting with 9f6230b902fbfcc927a9ec14c6c1e0ba1e51d2554a1dcb31e9a0291999764d39 not found: ID does not exist"
Jul 28 16:49:07.172273 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.172253    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" path="/var/lib/kubelet/pods/1e94e79f-271c-42f2-a910-eb4f95df0ba3/volumes"
Jul 28 16:49:07.904012 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.903976    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"]
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904403    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="storage-initializer"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904421    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="storage-initializer"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904433    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904440    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904450    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerName="storage-initializer"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904458    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerName="storage-initializer"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904468    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerName="storage-initializer"
Jul 28 16:49:07.904473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904477    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerName="storage-initializer"
Jul 28 16:49:07.904875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904490    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kube-rbac-proxy"
Jul 28 16:49:07.904875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904497    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kube-rbac-proxy"
Jul 28 16:49:07.904875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904581    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kserve-container"
Jul 28 16:49:07.904875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904595    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerName="storage-initializer"
Jul 28 16:49:07.904875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904604    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e94e79f-271c-42f2-a910-eb4f95df0ba3" containerName="kube-rbac-proxy"
Jul 28 16:49:07.904875 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.904737    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df8e4d27-5261-4b7b-be82-9a84d5065c6b" containerName="storage-initializer"
Jul 28 16:49:07.909267 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.909247    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:07.911862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.911840    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-1ebd71-predictor-serving-cert\""
Jul 28 16:49:07.911989 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.911862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-1ebd71\""
Jul 28 16:49:07.911989 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.911843    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:49:07.912119 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.912096    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-1ebd71-kube-rbac-proxy-sar-config\""
Jul 28 16:49:07.912175 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.912111    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-1ebd71-dockercfg-bmm2p\""
Jul 28 16:49:07.912755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.912733    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:49:07.912877 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.912806    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:49:07.923111 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:07.923092    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"]
Jul 28 16:49:08.040838 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.040812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07b7e90-df9d-47b0-ac91-1366923f71d0-kserve-provision-location\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.040960 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.040841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07b7e90-df9d-47b0-ac91-1366923f71d0-proxy-tls\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.040960 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.040875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-cabundle-cert\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.041035 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.040953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wws\" (UniqueName: \"kubernetes.io/projected/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-api-access-v7wws\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.041035 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.041013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.142314 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142287    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.142314 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07b7e90-df9d-47b0-ac91-1366923f71d0-kserve-provision-location\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.142503 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07b7e90-df9d-47b0-ac91-1366923f71d0-proxy-tls\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.142503 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-cabundle-cert\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.142503 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142441    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wws\" (UniqueName: \"kubernetes.io/projected/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-api-access-v7wws\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.142716 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142698    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07b7e90-df9d-47b0-ac91-1366923f71d0-kserve-provision-location\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.143004 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.142982    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.143069 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.143007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-cabundle-cert\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.144850 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.144834    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07b7e90-df9d-47b0-ac91-1366923f71d0-proxy-tls\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.152146 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.152122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wws\" (UniqueName: \"kubernetes.io/projected/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-api-access-v7wws\") pod \"isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") " pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.219352 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.219327    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:08.344764 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:08.344739    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"]
Jul 28 16:49:08.348013 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:49:08.347991    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07b7e90_df9d_47b0_ac91_1366923f71d0.slice/crio-61fb6b76f5aff32f0dbb943cf217b49336b6446cb056935df1b886bd34fea55a WatchSource:0}: Error finding container 61fb6b76f5aff32f0dbb943cf217b49336b6446cb056935df1b886bd34fea55a: Status 404 returned error can't find the container with id 61fb6b76f5aff32f0dbb943cf217b49336b6446cb056935df1b886bd34fea55a
Jul 28 16:49:09.063176 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:09.063144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" event={"ID":"c07b7e90-df9d-47b0-ac91-1366923f71d0","Type":"ContainerStarted","Data":"3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919"}
Jul 28 16:49:09.063552 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:09.063201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" event={"ID":"c07b7e90-df9d-47b0-ac91-1366923f71d0","Type":"ContainerStarted","Data":"61fb6b76f5aff32f0dbb943cf217b49336b6446cb056935df1b886bd34fea55a"}
Jul 28 16:49:15.089731 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:15.089704    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5_c07b7e90-df9d-47b0-ac91-1366923f71d0/storage-initializer/0.log"
Jul 28 16:49:15.090084 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:15.089748    2573 generic.go:356] "Generic (PLEG): container finished" podID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerID="3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919" exitCode=1
Jul 28 16:49:15.090084 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:15.089793    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" event={"ID":"c07b7e90-df9d-47b0-ac91-1366923f71d0","Type":"ContainerDied","Data":"3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919"}
Jul 28 16:49:16.095493 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:16.095467    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5_c07b7e90-df9d-47b0-ac91-1366923f71d0/storage-initializer/0.log"
Jul 28 16:49:16.095903 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:16.095529    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" event={"ID":"c07b7e90-df9d-47b0-ac91-1366923f71d0","Type":"ContainerStarted","Data":"e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570"}
Jul 28 16:49:17.876245 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:17.876214    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"]
Jul 28 16:49:17.876629 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:17.876449    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer" containerID="cri-o://e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570" gracePeriod=30
Jul 28 16:49:18.516870 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.516848    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5_c07b7e90-df9d-47b0-ac91-1366923f71d0/storage-initializer/1.log"
Jul 28 16:49:18.517212 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.517197    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5_c07b7e90-df9d-47b0-ac91-1366923f71d0/storage-initializer/0.log"
Jul 28 16:49:18.519299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.519284    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:18.619652 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.619569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07b7e90-df9d-47b0-ac91-1366923f71d0-proxy-tls\") pod \"c07b7e90-df9d-47b0-ac91-1366923f71d0\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") "
Jul 28 16:49:18.619652 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.619612    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-cabundle-cert\") pod \"c07b7e90-df9d-47b0-ac91-1366923f71d0\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") "
Jul 28 16:49:18.619652 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.619637    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07b7e90-df9d-47b0-ac91-1366923f71d0-kserve-provision-location\") pod \"c07b7e90-df9d-47b0-ac91-1366923f71d0\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") "
Jul 28 16:49:18.619943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.619690    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7wws\" (UniqueName: \"kubernetes.io/projected/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-api-access-v7wws\") pod \"c07b7e90-df9d-47b0-ac91-1366923f71d0\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") "
Jul 28 16:49:18.619943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.619712    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-rbac-proxy-sar-config\") pod \"c07b7e90-df9d-47b0-ac91-1366923f71d0\" (UID: \"c07b7e90-df9d-47b0-ac91-1366923f71d0\") "
Jul 28 16:49:18.619943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.619890    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07b7e90-df9d-47b0-ac91-1366923f71d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c07b7e90-df9d-47b0-ac91-1366923f71d0" (UID: "c07b7e90-df9d-47b0-ac91-1366923f71d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:49:18.620093 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.620046    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c07b7e90-df9d-47b0-ac91-1366923f71d0" (UID: "c07b7e90-df9d-47b0-ac91-1366923f71d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:49:18.620130 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.620107    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c07b7e90-df9d-47b0-ac91-1366923f71d0" (UID: "c07b7e90-df9d-47b0-ac91-1366923f71d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:49:18.621807 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.621780    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07b7e90-df9d-47b0-ac91-1366923f71d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c07b7e90-df9d-47b0-ac91-1366923f71d0" (UID: "c07b7e90-df9d-47b0-ac91-1366923f71d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:49:18.621912 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.621818    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-api-access-v7wws" (OuterVolumeSpecName: "kube-api-access-v7wws") pod "c07b7e90-df9d-47b0-ac91-1366923f71d0" (UID: "c07b7e90-df9d-47b0-ac91-1366923f71d0"). InnerVolumeSpecName "kube-api-access-v7wws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:49:18.721221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.721189    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wws\" (UniqueName: \"kubernetes.io/projected/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-api-access-v7wws\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:18.721221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.721212    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:18.721221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.721222    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c07b7e90-df9d-47b0-ac91-1366923f71d0-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:18.721463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.721233    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07b7e90-df9d-47b0-ac91-1366923f71d0-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:18.721463 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:18.721241    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07b7e90-df9d-47b0-ac91-1366923f71d0-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:49:19.107661 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.107634    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5_c07b7e90-df9d-47b0-ac91-1366923f71d0/storage-initializer/1.log"
Jul 28 16:49:19.108054 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.108039    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5_c07b7e90-df9d-47b0-ac91-1366923f71d0/storage-initializer/0.log"
Jul 28 16:49:19.108125 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.108072    2573 generic.go:356] "Generic (PLEG): container finished" podID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerID="e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570" exitCode=1
Jul 28 16:49:19.108125 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.108100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" event={"ID":"c07b7e90-df9d-47b0-ac91-1366923f71d0","Type":"ContainerDied","Data":"e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570"}
Jul 28 16:49:19.108198 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.108126    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"
Jul 28 16:49:19.108198 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.108184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5" event={"ID":"c07b7e90-df9d-47b0-ac91-1366923f71d0","Type":"ContainerDied","Data":"61fb6b76f5aff32f0dbb943cf217b49336b6446cb056935df1b886bd34fea55a"}
Jul 28 16:49:19.108267 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.108210    2573 scope.go:117] "RemoveContainer" containerID="e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570"
Jul 28 16:49:19.116777 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.116763    2573 scope.go:117] "RemoveContainer" containerID="3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919"
Jul 28 16:49:19.123823 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.123803    2573 scope.go:117] "RemoveContainer" containerID="e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570"
Jul 28 16:49:19.124079 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:49:19.124062    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570\": container with ID starting with e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570 not found: ID does not exist" containerID="e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570"
Jul 28 16:49:19.124117 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.124089    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570"} err="failed to get container status \"e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570\": rpc error: code = NotFound desc = could not find container \"e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570\": container with ID starting with e6707c14e876e31e5d71d84cc62f5d8809574c5a486dea81f2c5765ad65c1570 not found: ID does not exist"
Jul 28 16:49:19.124117 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.124104    2573 scope.go:117] "RemoveContainer" containerID="3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919"
Jul 28 16:49:19.124330 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:49:19.124314    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919\": container with ID starting with 3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919 not found: ID does not exist" containerID="3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919"
Jul 28 16:49:19.124365 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.124337    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919"} err="failed to get container status \"3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919\": rpc error: code = NotFound desc = could not find container \"3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919\": container with ID starting with 3c67bbf5cdc1919dac004090a0e45552544fe645d44ee9066433d29721f7f919 not found: ID does not exist"
Jul 28 16:49:19.143712 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.143685    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"]
Jul 28 16:49:19.148044 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.148024    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-1ebd71-predictor-77998b7f76-d6bp5"]
Jul 28 16:49:19.172806 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:19.172782    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" path="/var/lib/kubelet/pods/c07b7e90-df9d-47b0-ac91-1366923f71d0/volumes"
Jul 28 16:49:23.132933 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.132903    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"]
Jul 28 16:49:23.133434 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.133295    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer"
Jul 28 16:49:23.133434 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.133311    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer"
Jul 28 16:49:23.133434 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.133322    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer"
Jul 28 16:49:23.133434 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.133332    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer"
Jul 28 16:49:23.133656 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.133440    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer"
Jul 28 16:49:23.133656 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.133573    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07b7e90-df9d-47b0-ac91-1366923f71d0" containerName="storage-initializer"
Jul 28 16:49:23.141646 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.141623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.144566 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.144543    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:49:23.144804 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.144791    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:49:23.144895 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.144873    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:49:23.145321 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.145296    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:49:23.145435 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.145333    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-3c435-predictor-serving-cert\""
Jul 28 16:49:23.145435 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.145296    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:49:23.145435 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.145302    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-3c435-kube-rbac-proxy-sar-config\""
Jul 28 16:49:23.148204 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.148183    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"]
Jul 28 16:49:23.253604 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.253575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kserve-provision-location\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.253604 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.253610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.253806 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.253631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbrc\" (UniqueName: \"kubernetes.io/projected/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-api-access-frbrc\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.253806 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.253720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-cabundle-cert\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.253806 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.253772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f57fff-28a3-4087-9aec-b48326d7c4f0-proxy-tls\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.354616 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.354580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kserve-provision-location\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.354616 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.354620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.354832 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.354646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbrc\" (UniqueName: \"kubernetes.io/projected/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-api-access-frbrc\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.354832 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.354675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-cabundle-cert\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.354832 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.354707    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f57fff-28a3-4087-9aec-b48326d7c4f0-proxy-tls\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.355004 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.354984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kserve-provision-location\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.355321 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.355299    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.355459 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.355444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-cabundle-cert\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.357126 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.357111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f57fff-28a3-4087-9aec-b48326d7c4f0-proxy-tls\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.364621 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.364602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbrc\" (UniqueName: \"kubernetes.io/projected/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-api-access-frbrc\") pod \"raw-sklearn-3c435-predictor-864444488d-74727\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") " pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.452859 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.452802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:23.581275 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:23.581229    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"]
Jul 28 16:49:23.583815 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:49:23.583779    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f57fff_28a3_4087_9aec_b48326d7c4f0.slice/crio-9187d662f1ba49b72b3f66aa0784b88550abc96169b1b76346b2cc57f0f99e3b WatchSource:0}: Error finding container 9187d662f1ba49b72b3f66aa0784b88550abc96169b1b76346b2cc57f0f99e3b: Status 404 returned error can't find the container with id 9187d662f1ba49b72b3f66aa0784b88550abc96169b1b76346b2cc57f0f99e3b
Jul 28 16:49:24.130139 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:24.130105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerStarted","Data":"8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4"}
Jul 28 16:49:24.130285 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:24.130159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerStarted","Data":"9187d662f1ba49b72b3f66aa0784b88550abc96169b1b76346b2cc57f0f99e3b"}
Jul 28 16:49:28.147087 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:28.147055    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerID="8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4" exitCode=0
Jul 28 16:49:28.147520 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:28.147128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerDied","Data":"8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4"}
Jul 28 16:49:29.152851 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:29.152820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerStarted","Data":"25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b"}
Jul 28 16:49:29.153231 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:29.152875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerStarted","Data":"ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d"}
Jul 28 16:49:29.153231 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:29.153044    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:29.176339 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:29.176264    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podStartSLOduration=2.387973938 podStartE2EDuration="6.176246712s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788272774s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:49:29.174468987 +0000 UTC m=+1754.567785526" watchObservedRunningTime="2026-07-28 16:49:29.176246712 +0000 UTC m=+1754.569563253"
Jul 28 16:49:30.157304 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:30.157275    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:30.158539 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:30.158514    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:49:31.161112 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:31.161075    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:49:36.165089 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:36.165057    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:49:36.165605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:36.165582    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:49:46.166241 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:46.166201    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:49:56.165652 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:49:56.165611    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:50:06.166183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:06.166100    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:50:15.124874 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:15.124840    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:50:15.128068 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:15.128040    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:50:16.166096 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:16.166062    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:50:26.165588 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:26.165547    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:50:36.166039 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:36.166011    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:50:43.246536 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:43.246500    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"]
Jul 28 16:50:43.247415 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:43.247388    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" containerID="cri-o://ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d" gracePeriod=30
Jul 28 16:50:43.248065 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:43.247785    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kube-rbac-proxy" containerID="cri-o://25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b" gracePeriod=30
Jul 28 16:50:43.427862 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:43.427831    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerID="25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b" exitCode=2
Jul 28 16:50:43.428012 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:43.427865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerDied","Data":"25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b"}
Jul 28 16:50:46.161266 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:46.161228    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 16:50:46.165527 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:46.165503    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:50:47.184473 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.184451    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:50:47.255783 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.255722    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f57fff-28a3-4087-9aec-b48326d7c4f0-proxy-tls\") pod \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") "
Jul 28 16:50:47.255783 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.255767    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kserve-provision-location\") pod \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") "
Jul 28 16:50:47.256015 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.255832    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-rbac-proxy-sar-config\") pod \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") "
Jul 28 16:50:47.256015 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.255849    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frbrc\" (UniqueName: \"kubernetes.io/projected/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-api-access-frbrc\") pod \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") "
Jul 28 16:50:47.256015 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.255884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-cabundle-cert\") pod \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\" (UID: \"a2f57fff-28a3-4087-9aec-b48326d7c4f0\") "
Jul 28 16:50:47.256280 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.256135    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2f57fff-28a3-4087-9aec-b48326d7c4f0" (UID: "a2f57fff-28a3-4087-9aec-b48326d7c4f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:50:47.256638 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.256515    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2f57fff-28a3-4087-9aec-b48326d7c4f0" (UID: "a2f57fff-28a3-4087-9aec-b48326d7c4f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:50:47.256746 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.256664    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2f57fff-28a3-4087-9aec-b48326d7c4f0" (UID: "a2f57fff-28a3-4087-9aec-b48326d7c4f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:50:47.258072 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.258043    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-api-access-frbrc" (OuterVolumeSpecName: "kube-api-access-frbrc") pod "a2f57fff-28a3-4087-9aec-b48326d7c4f0" (UID: "a2f57fff-28a3-4087-9aec-b48326d7c4f0"). InnerVolumeSpecName "kube-api-access-frbrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:50:47.258170 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.258133    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f57fff-28a3-4087-9aec-b48326d7c4f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2f57fff-28a3-4087-9aec-b48326d7c4f0" (UID: "a2f57fff-28a3-4087-9aec-b48326d7c4f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:50:47.357151 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.357129    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:50:47.357151 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.357148    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frbrc\" (UniqueName: \"kubernetes.io/projected/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kube-api-access-frbrc\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:50:47.357277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.357157    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2f57fff-28a3-4087-9aec-b48326d7c4f0-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:50:47.357277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.357166    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f57fff-28a3-4087-9aec-b48326d7c4f0-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:50:47.357277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.357174    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f57fff-28a3-4087-9aec-b48326d7c4f0-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:50:47.444186 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.444156    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerID="ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d" exitCode=0
Jul 28 16:50:47.444369 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.444209    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"
Jul 28 16:50:47.444369 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.444245    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerDied","Data":"ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d"}
Jul 28 16:50:47.444369 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.444323    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727" event={"ID":"a2f57fff-28a3-4087-9aec-b48326d7c4f0","Type":"ContainerDied","Data":"9187d662f1ba49b72b3f66aa0784b88550abc96169b1b76346b2cc57f0f99e3b"}
Jul 28 16:50:47.444369 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.444360    2573 scope.go:117] "RemoveContainer" containerID="25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b"
Jul 28 16:50:47.452829 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.452814    2573 scope.go:117] "RemoveContainer" containerID="ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d"
Jul 28 16:50:47.459990 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.459975    2573 scope.go:117] "RemoveContainer" containerID="8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4"
Jul 28 16:50:47.465427 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.465360    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"]
Jul 28 16:50:47.467648 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.467633    2573 scope.go:117] "RemoveContainer" containerID="25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b"
Jul 28 16:50:47.467910 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:50:47.467895    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b\": container with ID starting with 25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b not found: ID does not exist" containerID="25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b"
Jul 28 16:50:47.467956 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.467918    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b"} err="failed to get container status \"25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b\": rpc error: code = NotFound desc = could not find container \"25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b\": container with ID starting with 25b619bd3c8c2abeb1f7f204b4b360f937f746dcce27e5ff8fbeb3fbdb97e86b not found: ID does not exist"
Jul 28 16:50:47.467956 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.467932    2573 scope.go:117] "RemoveContainer" containerID="ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d"
Jul 28 16:50:47.468172 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:50:47.468156    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d\": container with ID starting with ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d not found: ID does not exist" containerID="ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d"
Jul 28 16:50:47.468212 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.468191    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d"} err="failed to get container status \"ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d\": rpc error: code = NotFound desc = could not find container \"ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d\": container with ID starting with ac91d509158ef7a287da4cd2fa360e071261fb7d790623def7c5865d8f30448d not found: ID does not exist"
Jul 28 16:50:47.468212 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.468206    2573 scope.go:117] "RemoveContainer" containerID="8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4"
Jul 28 16:50:47.468463 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:50:47.468448    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4\": container with ID starting with 8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4 not found: ID does not exist" containerID="8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4"
Jul 28 16:50:47.468509 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.468467    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4"} err="failed to get container status \"8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4\": rpc error: code = NotFound desc = could not find container \"8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4\": container with ID starting with 8b1e06f8d5e21d0a44dd392fcff6e7fd7650eda0b17a14b7d7e6f5ae4f6fc2c4 not found: ID does not exist"
Jul 28 16:50:47.470980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:47.470962    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3c435-predictor-864444488d-74727"]
Jul 28 16:50:48.512368 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512334    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"]
Jul 28 16:50:48.512738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512661    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="storage-initializer"
Jul 28 16:50:48.512738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512675    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="storage-initializer"
Jul 28 16:50:48.512738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512695    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kube-rbac-proxy"
Jul 28 16:50:48.512738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512701    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kube-rbac-proxy"
Jul 28 16:50:48.512738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512709    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container"
Jul 28 16:50:48.512738 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container"
Jul 28 16:50:48.512925 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512762    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kserve-container"
Jul 28 16:50:48.512925 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.512771    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" containerName="kube-rbac-proxy"
Jul 28 16:50:48.517345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.517325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.520799 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.520781    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jlwpj\""
Jul 28 16:50:48.521252 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.521223    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-f5110-predictor-serving-cert\""
Jul 28 16:50:48.521632 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.521612    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:50:48.521793 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.521775    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 16:50:48.521916 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.521902    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:50:48.522099 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.522087    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 16:50:48.522760 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.522740    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-f5110-kube-rbac-proxy-sar-config\""
Jul 28 16:50:48.528549 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.528529    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"]
Jul 28 16:50:48.666904 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.666879    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e5e65a-0df2-4e20-9758-24f976bcbf07-proxy-tls\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.667033 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.666919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.667033 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.666972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-cabundle-cert\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.667033 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.667009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8v6\" (UniqueName: \"kubernetes.io/projected/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-api-access-zn8v6\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.667176 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.667085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e5e65a-0df2-4e20-9758-24f976bcbf07-kserve-provision-location\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768195 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e5e65a-0df2-4e20-9758-24f976bcbf07-proxy-tls\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768195 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768353 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-cabundle-cert\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768353 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8v6\" (UniqueName: \"kubernetes.io/projected/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-api-access-zn8v6\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768353 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e5e65a-0df2-4e20-9758-24f976bcbf07-kserve-provision-location\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768719 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768698    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e5e65a-0df2-4e20-9758-24f976bcbf07-kserve-provision-location\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.768943 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-cabundle-cert\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.769007 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.768966    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.770636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.770619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e5e65a-0df2-4e20-9758-24f976bcbf07-proxy-tls\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.777345 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.777325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8v6\" (UniqueName: \"kubernetes.io/projected/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-api-access-zn8v6\") pod \"raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.827971 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.827950    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:48.955131 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:48.955078    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"]
Jul 28 16:50:48.958857 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:50:48.958833    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e5e65a_0df2_4e20_9758_24f976bcbf07.slice/crio-f10ebb2d21f7c998c192bd5bac1ad6c5adbbf2bb622a1b38e912797975fb536d WatchSource:0}: Error finding container f10ebb2d21f7c998c192bd5bac1ad6c5adbbf2bb622a1b38e912797975fb536d: Status 404 returned error can't find the container with id f10ebb2d21f7c998c192bd5bac1ad6c5adbbf2bb622a1b38e912797975fb536d
Jul 28 16:50:49.172897 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:49.172820    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f57fff-28a3-4087-9aec-b48326d7c4f0" path="/var/lib/kubelet/pods/a2f57fff-28a3-4087-9aec-b48326d7c4f0/volumes"
Jul 28 16:50:49.453660 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:49.453573    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerStarted","Data":"5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df"}
Jul 28 16:50:49.453660 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:49.453641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerStarted","Data":"f10ebb2d21f7c998c192bd5bac1ad6c5adbbf2bb622a1b38e912797975fb536d"}
Jul 28 16:50:53.469431 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:53.469398    2573 generic.go:356] "Generic (PLEG): container finished" podID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerID="5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df" exitCode=0
Jul 28 16:50:53.469789 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:53.469460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerDied","Data":"5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df"}
Jul 28 16:50:54.474896 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:54.474866    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerStarted","Data":"8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706"}
Jul 28 16:50:54.475253 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:54.474924    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerStarted","Data":"c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7"}
Jul 28 16:50:54.475253 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:54.475096    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:54.495300 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:54.495256    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podStartSLOduration=2.556612472 podStartE2EDuration="6.495242236s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.938629764s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:50:54.494468945 +0000 UTC m=+1839.887785487" watchObservedRunningTime="2026-07-28 16:50:54.495242236 +0000 UTC m=+1839.888558829"
Jul 28 16:50:55.479097 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:55.479073    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:50:55.480013 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:55.479989    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:50:56.482726 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:50:56.482692    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:51:01.486774 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:01.486746    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:51:01.487393 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:01.487345    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:51:11.487553 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:11.487513    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:51:21.487861 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:21.487817    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:51:31.487949 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:31.487915    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:51:41.487641 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:41.487563    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:51:51.487736 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:51:51.487702    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:52:01.488183 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:01.488156    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:52:08.514568 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:08.514531    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"]
Jul 28 16:52:08.514975 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:08.514874    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" containerID="cri-o://c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7" gracePeriod=30
Jul 28 16:52:08.515046 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:08.514951    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kube-rbac-proxy" containerID="cri-o://8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706" gracePeriod=30
Jul 28 16:52:08.766198 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:08.766127    2573 generic.go:356] "Generic (PLEG): container finished" podID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerID="8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706" exitCode=2
Jul 28 16:52:08.766317 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:08.766196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerDied","Data":"8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706"}
Jul 28 16:52:11.483498 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:11.483453    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 16:52:11.487725 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:11.487702    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:52:12.646700 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.646676    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:52:12.762650 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.762582    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e5e65a-0df2-4e20-9758-24f976bcbf07-proxy-tls\") pod \"24e5e65a-0df2-4e20-9758-24f976bcbf07\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") "
Jul 28 16:52:12.762650 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.762632    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-cabundle-cert\") pod \"24e5e65a-0df2-4e20-9758-24f976bcbf07\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") "
Jul 28 16:52:12.762855 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.762676    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e5e65a-0df2-4e20-9758-24f976bcbf07-kserve-provision-location\") pod \"24e5e65a-0df2-4e20-9758-24f976bcbf07\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") "
Jul 28 16:52:12.762855 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.762702    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-rbac-proxy-sar-config\") pod \"24e5e65a-0df2-4e20-9758-24f976bcbf07\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") "
Jul 28 16:52:12.762855 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.762724    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn8v6\" (UniqueName: \"kubernetes.io/projected/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-api-access-zn8v6\") pod \"24e5e65a-0df2-4e20-9758-24f976bcbf07\" (UID: \"24e5e65a-0df2-4e20-9758-24f976bcbf07\") "
Jul 28 16:52:12.763024 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.762997    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e5e65a-0df2-4e20-9758-24f976bcbf07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24e5e65a-0df2-4e20-9758-24f976bcbf07" (UID: "24e5e65a-0df2-4e20-9758-24f976bcbf07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:52:12.763061 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.763026    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24e5e65a-0df2-4e20-9758-24f976bcbf07" (UID: "24e5e65a-0df2-4e20-9758-24f976bcbf07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:52:12.763061 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.763022    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24e5e65a-0df2-4e20-9758-24f976bcbf07" (UID: "24e5e65a-0df2-4e20-9758-24f976bcbf07"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:52:12.764760 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.764739    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e5e65a-0df2-4e20-9758-24f976bcbf07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24e5e65a-0df2-4e20-9758-24f976bcbf07" (UID: "24e5e65a-0df2-4e20-9758-24f976bcbf07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:52:12.764863 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.764795    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-api-access-zn8v6" (OuterVolumeSpecName: "kube-api-access-zn8v6") pod "24e5e65a-0df2-4e20-9758-24f976bcbf07" (UID: "24e5e65a-0df2-4e20-9758-24f976bcbf07"). InnerVolumeSpecName "kube-api-access-zn8v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:52:12.783304 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.783280    2573 generic.go:356] "Generic (PLEG): container finished" podID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerID="c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7" exitCode=0
Jul 28 16:52:12.783418 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.783330    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"
Jul 28 16:52:12.783418 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.783337    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerDied","Data":"c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7"}
Jul 28 16:52:12.783503 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.783419    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z" event={"ID":"24e5e65a-0df2-4e20-9758-24f976bcbf07","Type":"ContainerDied","Data":"f10ebb2d21f7c998c192bd5bac1ad6c5adbbf2bb622a1b38e912797975fb536d"}
Jul 28 16:52:12.783503 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.783459    2573 scope.go:117] "RemoveContainer" containerID="8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706"
Jul 28 16:52:12.795555 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.795529    2573 scope.go:117] "RemoveContainer" containerID="c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7"
Jul 28 16:52:12.802631 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.802615    2573 scope.go:117] "RemoveContainer" containerID="5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df"
Jul 28 16:52:12.809389 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.809353    2573 scope.go:117] "RemoveContainer" containerID="8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706"
Jul 28 16:52:12.809629 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:52:12.809610    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706\": container with ID starting with 8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706 not found: ID does not exist" containerID="8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706"
Jul 28 16:52:12.809681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.809638    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706"} err="failed to get container status \"8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706\": rpc error: code = NotFound desc = could not find container \"8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706\": container with ID starting with 8de230763a98abbf173e65f1c2e4287095fd9d27ed7b8d7d4287c28af6b0f706 not found: ID does not exist"
Jul 28 16:52:12.809681 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.809654    2573 scope.go:117] "RemoveContainer" containerID="c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7"
Jul 28 16:52:12.809879 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:52:12.809862    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7\": container with ID starting with c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7 not found: ID does not exist" containerID="c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7"
Jul 28 16:52:12.809924 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.809890    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7"} err="failed to get container status \"c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7\": rpc error: code = NotFound desc = could not find container \"c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7\": container with ID starting with c1556771451b466038bfda3fe45b1f6b2e2217bd40e01cbda67595f9481a19c7 not found: ID does not exist"
Jul 28 16:52:12.809924 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.809906    2573 scope.go:117] "RemoveContainer" containerID="5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df"
Jul 28 16:52:12.810108 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:52:12.810093    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df\": container with ID starting with 5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df not found: ID does not exist" containerID="5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df"
Jul 28 16:52:12.810144 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.810114    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df"} err="failed to get container status \"5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df\": rpc error: code = NotFound desc = could not find container \"5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df\": container with ID starting with 5184630d5c8d4ec0e070a54aa6d100ca74b77514512e400f9b9721d085c7d5df not found: ID does not exist"
Jul 28 16:52:12.824804 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.824784    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"]
Jul 28 16:52:12.832416 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.832397    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5110-predictor-8655474ffb-gkd4z"]
Jul 28 16:52:12.863367 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.863347    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24e5e65a-0df2-4e20-9758-24f976bcbf07-proxy-tls\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:12.863367 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.863364    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-cabundle-cert\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:12.863496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.863389    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24e5e65a-0df2-4e20-9758-24f976bcbf07-kserve-provision-location\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:12.863496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.863399    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:12.863496 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:12.863407    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8v6\" (UniqueName: \"kubernetes.io/projected/24e5e65a-0df2-4e20-9758-24f976bcbf07-kube-api-access-zn8v6\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:13.171779 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:13.171720    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" path="/var/lib/kubelet/pods/24e5e65a-0df2-4e20-9758-24f976bcbf07/volumes"
Jul 28 16:52:15.000923 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.000887    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9m7gh/must-gather-4clfn"]
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001150    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001160    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001173    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="storage-initializer"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001178    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="storage-initializer"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001191    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kube-rbac-proxy"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001196    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kube-rbac-proxy"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001251    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kserve-container"
Jul 28 16:52:15.001410 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.001261    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e5e65a-0df2-4e20-9758-24f976bcbf07" containerName="kube-rbac-proxy"
Jul 28 16:52:15.005715 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.005691    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.008092 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.008071    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9m7gh\"/\"openshift-service-ca.crt\""
Jul 28 16:52:15.008567 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.008551    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9m7gh\"/\"default-dockercfg-vlhdd\""
Jul 28 16:52:15.008567 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.008562    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9m7gh\"/\"kube-root-ca.crt\""
Jul 28 16:52:15.012954 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.012933    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9m7gh/must-gather-4clfn"]
Jul 28 16:52:15.078602 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.078569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/013c910a-3d55-4b5b-91b5-622957bb6363-must-gather-output\") pod \"must-gather-4clfn\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") " pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.078602 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.078603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg57h\" (UniqueName: \"kubernetes.io/projected/013c910a-3d55-4b5b-91b5-622957bb6363-kube-api-access-hg57h\") pod \"must-gather-4clfn\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") " pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.179891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.179868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/013c910a-3d55-4b5b-91b5-622957bb6363-must-gather-output\") pod \"must-gather-4clfn\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") " pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.180018 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.179897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg57h\" (UniqueName: \"kubernetes.io/projected/013c910a-3d55-4b5b-91b5-622957bb6363-kube-api-access-hg57h\") pod \"must-gather-4clfn\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") " pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.180153 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.180136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/013c910a-3d55-4b5b-91b5-622957bb6363-must-gather-output\") pod \"must-gather-4clfn\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") " pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.188324 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.188305    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg57h\" (UniqueName: \"kubernetes.io/projected/013c910a-3d55-4b5b-91b5-622957bb6363-kube-api-access-hg57h\") pod \"must-gather-4clfn\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") " pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.325656 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.325633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:15.450678 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.450639    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9m7gh/must-gather-4clfn"]
Jul 28 16:52:15.453425 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:52:15.453395    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod013c910a_3d55_4b5b_91b5_622957bb6363.slice/crio-805fc6561215355c672da1e85b18bf451d4f545a86b0e1f2390d28339f230e37 WatchSource:0}: Error finding container 805fc6561215355c672da1e85b18bf451d4f545a86b0e1f2390d28339f230e37: Status 404 returned error can't find the container with id 805fc6561215355c672da1e85b18bf451d4f545a86b0e1f2390d28339f230e37
Jul 28 16:52:15.455362 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.455347    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:52:15.796771 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:15.796735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9m7gh/must-gather-4clfn" event={"ID":"013c910a-3d55-4b5b-91b5-622957bb6363","Type":"ContainerStarted","Data":"805fc6561215355c672da1e85b18bf451d4f545a86b0e1f2390d28339f230e37"}
Jul 28 16:52:21.827986 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:21.827955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9m7gh/must-gather-4clfn" event={"ID":"013c910a-3d55-4b5b-91b5-622957bb6363","Type":"ContainerStarted","Data":"eeb1682f6f1f32360502d4343e489e00622b9e9e946e7b559a6f80fec8886078"}
Jul 28 16:52:22.835871 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:22.835833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9m7gh/must-gather-4clfn" event={"ID":"013c910a-3d55-4b5b-91b5-622957bb6363","Type":"ContainerStarted","Data":"e9ae8b3f17a002a45669939294a7d9afedd92f397a6929c32880ec31a31aea38"}
Jul 28 16:52:22.853028 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:22.852987    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9m7gh/must-gather-4clfn" podStartSLOduration=2.703902529 podStartE2EDuration="8.852973119s" totalImagesPullingTime="6.14907059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:52:22.851658492 +0000 UTC m=+1928.244975030" watchObservedRunningTime="2026-07-28 16:52:22.852973119 +0000 UTC m=+1928.246289656"
Jul 28 16:52:43.924177 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:43.924142    2573 generic.go:356] "Generic (PLEG): container finished" podID="013c910a-3d55-4b5b-91b5-622957bb6363" containerID="eeb1682f6f1f32360502d4343e489e00622b9e9e946e7b559a6f80fec8886078" exitCode=0
Jul 28 16:52:43.924600 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:43.924215    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9m7gh/must-gather-4clfn" event={"ID":"013c910a-3d55-4b5b-91b5-622957bb6363","Type":"ContainerDied","Data":"eeb1682f6f1f32360502d4343e489e00622b9e9e946e7b559a6f80fec8886078"}
Jul 28 16:52:43.924600 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:43.924538    2573 scope.go:117] "RemoveContainer" containerID="eeb1682f6f1f32360502d4343e489e00622b9e9e946e7b559a6f80fec8886078"
Jul 28 16:52:44.163058 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.163035    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9m7gh_must-gather-4clfn_013c910a-3d55-4b5b-91b5-622957bb6363/gather/0.log"
Jul 28 16:52:44.759945 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.759920    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kvj4s/must-gather-4hfcq"]
Jul 28 16:52:44.762497 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.762480    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:44.764641 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.764596    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvj4s\"/\"openshift-service-ca.crt\""
Jul 28 16:52:44.764641 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.764616    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kvj4s\"/\"kube-root-ca.crt\""
Jul 28 16:52:44.764834 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.764757    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kvj4s\"/\"default-dockercfg-nzlck\""
Jul 28 16:52:44.770202 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.770180    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvj4s/must-gather-4hfcq"]
Jul 28 16:52:44.816633 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.816611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpz2\" (UniqueName: \"kubernetes.io/projected/ffedb3bd-588c-4d1f-9a55-1861ec089892-kube-api-access-xhpz2\") pod \"must-gather-4hfcq\" (UID: \"ffedb3bd-588c-4d1f-9a55-1861ec089892\") " pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:44.816763 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.816667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffedb3bd-588c-4d1f-9a55-1861ec089892-must-gather-output\") pod \"must-gather-4hfcq\" (UID: \"ffedb3bd-588c-4d1f-9a55-1861ec089892\") " pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:44.917971 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.917946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffedb3bd-588c-4d1f-9a55-1861ec089892-must-gather-output\") pod \"must-gather-4hfcq\" (UID: \"ffedb3bd-588c-4d1f-9a55-1861ec089892\") " pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:44.918065 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.917983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpz2\" (UniqueName: \"kubernetes.io/projected/ffedb3bd-588c-4d1f-9a55-1861ec089892-kube-api-access-xhpz2\") pod \"must-gather-4hfcq\" (UID: \"ffedb3bd-588c-4d1f-9a55-1861ec089892\") " pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:44.918305 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.918289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffedb3bd-588c-4d1f-9a55-1861ec089892-must-gather-output\") pod \"must-gather-4hfcq\" (UID: \"ffedb3bd-588c-4d1f-9a55-1861ec089892\") " pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:44.925931 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:44.925908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpz2\" (UniqueName: \"kubernetes.io/projected/ffedb3bd-588c-4d1f-9a55-1861ec089892-kube-api-access-xhpz2\") pod \"must-gather-4hfcq\" (UID: \"ffedb3bd-588c-4d1f-9a55-1861ec089892\") " pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:45.071532 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:45.071475    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvj4s/must-gather-4hfcq"
Jul 28 16:52:45.196912 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:45.196892    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvj4s/must-gather-4hfcq"]
Jul 28 16:52:45.199320 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:52:45.199286    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffedb3bd_588c_4d1f_9a55_1861ec089892.slice/crio-d2cc36779dd5f84b2d1f6bd014f7043ece16bd5213fda5f06f6dcf89515e975a WatchSource:0}: Error finding container d2cc36779dd5f84b2d1f6bd014f7043ece16bd5213fda5f06f6dcf89515e975a: Status 404 returned error can't find the container with id d2cc36779dd5f84b2d1f6bd014f7043ece16bd5213fda5f06f6dcf89515e975a
Jul 28 16:52:45.934128 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:45.934090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvj4s/must-gather-4hfcq" event={"ID":"ffedb3bd-588c-4d1f-9a55-1861ec089892","Type":"ContainerStarted","Data":"d2cc36779dd5f84b2d1f6bd014f7043ece16bd5213fda5f06f6dcf89515e975a"}
Jul 28 16:52:46.942073 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:46.942031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvj4s/must-gather-4hfcq" event={"ID":"ffedb3bd-588c-4d1f-9a55-1861ec089892","Type":"ContainerStarted","Data":"57eb645ed1876402cbe8087866a7e4e91d0f338b031fb38f9cb78c79845fda5a"}
Jul 28 16:52:46.942597 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:46.942090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvj4s/must-gather-4hfcq" event={"ID":"ffedb3bd-588c-4d1f-9a55-1861ec089892","Type":"ContainerStarted","Data":"bbcf8f5db75ded1068f603996ab8fd99c1ee9c85161cb73558ac70588ac713fa"}
Jul 28 16:52:46.965643 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:46.965594    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kvj4s/must-gather-4hfcq" podStartSLOduration=2.195963984 podStartE2EDuration="2.965580068s" totalImagesPullingTime="769.616084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:52:46.964216627 +0000 UTC m=+1952.357533165" watchObservedRunningTime="2026-07-28 16:52:46.965580068 +0000 UTC m=+1952.358896606"
Jul 28 16:52:47.460834 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:47.460796    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zx225_cb496c43-a63c-4e6c-9cd7-1c74d7009d2c/global-pull-secret-syncer/0.log"
Jul 28 16:52:47.583447 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:47.583418    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wnvwk_07640db5-ab1c-4797-870b-4b9f3767dbe1/konnectivity-agent/0.log"
Jul 28 16:52:47.661570 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:47.661536    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-120.ec2.internal_a87de93e34c8837496bc8bb7c2576ca1/haproxy/0.log"
Jul 28 16:52:49.589694 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.589649    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-9m7gh/must-gather-4clfn"]
Jul 28 16:52:49.590268 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.589933    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-9m7gh/must-gather-4clfn" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="copy" containerID="cri-o://e9ae8b3f17a002a45669939294a7d9afedd92f397a6929c32880ec31a31aea38" gracePeriod=2
Jul 28 16:52:49.595751 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.595716    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-9m7gh/must-gather-4clfn"]
Jul 28 16:52:49.968657 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.966628    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9m7gh_must-gather-4clfn_013c910a-3d55-4b5b-91b5-622957bb6363/copy/0.log"
Jul 28 16:52:49.968657 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.967489    2573 generic.go:356] "Generic (PLEG): container finished" podID="013c910a-3d55-4b5b-91b5-622957bb6363" containerID="e9ae8b3f17a002a45669939294a7d9afedd92f397a6929c32880ec31a31aea38" exitCode=143
Jul 28 16:52:49.976026 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.974173    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805fc6561215355c672da1e85b18bf451d4f545a86b0e1f2390d28339f230e37"
Jul 28 16:52:49.995394 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.993524    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9m7gh_must-gather-4clfn_013c910a-3d55-4b5b-91b5-622957bb6363/copy/0.log"
Jul 28 16:52:49.998141 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:49.996046    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:49.998277 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:52:49.998154    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4clfn\" is forbidden: User \"system:node:ip-10-0-133-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9m7gh\": no relationship found between node 'ip-10-0-133-120.ec2.internal' and this object" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:50.079023 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.078980    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg57h\" (UniqueName: \"kubernetes.io/projected/013c910a-3d55-4b5b-91b5-622957bb6363-kube-api-access-hg57h\") pod \"013c910a-3d55-4b5b-91b5-622957bb6363\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") "
Jul 28 16:52:50.079214 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.079040    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/013c910a-3d55-4b5b-91b5-622957bb6363-must-gather-output\") pod \"013c910a-3d55-4b5b-91b5-622957bb6363\" (UID: \"013c910a-3d55-4b5b-91b5-622957bb6363\") "
Jul 28 16:52:50.080626 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.080591    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013c910a-3d55-4b5b-91b5-622957bb6363-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "013c910a-3d55-4b5b-91b5-622957bb6363" (UID: "013c910a-3d55-4b5b-91b5-622957bb6363"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:52:50.083636 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.083599    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013c910a-3d55-4b5b-91b5-622957bb6363-kube-api-access-hg57h" (OuterVolumeSpecName: "kube-api-access-hg57h") pod "013c910a-3d55-4b5b-91b5-622957bb6363" (UID: "013c910a-3d55-4b5b-91b5-622957bb6363"). InnerVolumeSpecName "kube-api-access-hg57h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:52:50.179777 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.179715    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg57h\" (UniqueName: \"kubernetes.io/projected/013c910a-3d55-4b5b-91b5-622957bb6363-kube-api-access-hg57h\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:50.179777 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.179749    2573 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/013c910a-3d55-4b5b-91b5-622957bb6363-must-gather-output\") on node \"ip-10-0-133-120.ec2.internal\" DevicePath \"\""
Jul 28 16:52:50.974034 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:50.974003    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:50.980653 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:52:50.980612    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4clfn\" is forbidden: User \"system:node:ip-10-0-133-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9m7gh\": no relationship found between node 'ip-10-0-133-120.ec2.internal' and this object" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:50.988106 ip-10-0-133-120 kubenswrapper[2573]: E0728 16:52:50.988068    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4clfn\" is forbidden: User \"system:node:ip-10-0-133-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9m7gh\": no relationship found between node 'ip-10-0-133-120.ec2.internal' and this object" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" pod="openshift-must-gather-9m7gh/must-gather-4clfn"
Jul 28 16:52:51.174765 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:51.174733    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" path="/var/lib/kubelet/pods/013c910a-3d55-4b5b-91b5-622957bb6363/volumes"
Jul 28 16:52:51.910771 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:51.910741    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-2hsb5_09bd4ee5-6f10-426f-b022-24ae529e264f/monitoring-plugin/0.log"
Jul 28 16:52:51.951153 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:51.951118    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njfjh_bf71d79a-7ffe-470d-b6ee-45b2c1e221cf/node-exporter/0.log"
Jul 28 16:52:51.977891 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:51.977811    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njfjh_bf71d79a-7ffe-470d-b6ee-45b2c1e221cf/kube-rbac-proxy/0.log"
Jul 28 16:52:52.006855 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.006826    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njfjh_bf71d79a-7ffe-470d-b6ee-45b2c1e221cf/init-textfile/0.log"
Jul 28 16:52:52.335173 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.335100    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/prometheus/0.log"
Jul 28 16:52:52.363850 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.363821    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/config-reloader/0.log"
Jul 28 16:52:52.394910 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.394877    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/thanos-sidecar/0.log"
Jul 28 16:52:52.433988 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.433941    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/kube-rbac-proxy-web/0.log"
Jul 28 16:52:52.466277 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.466241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/kube-rbac-proxy/0.log"
Jul 28 16:52:52.492568 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.492544    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/kube-rbac-proxy-thanos/0.log"
Jul 28 16:52:52.522314 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.522285    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe7b845-c895-4a3c-b7d3-bbef1aa9db1a/init-config-reloader/0.log"
Jul 28 16:52:52.562981 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.562948    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-9cczv_8f0cac8e-0b86-4709-b5af-d43d1b8813fa/prometheus-operator/0.log"
Jul 28 16:52:52.590605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.590523    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-9cczv_8f0cac8e-0b86-4709-b5af-d43d1b8813fa/kube-rbac-proxy/0.log"
Jul 28 16:52:52.746481 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.746393    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555b4fff79-gxwmp_da067996-34f2-4510-9dd1-4a5879f585ef/thanos-query/0.log"
Jul 28 16:52:52.773650 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.773625    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555b4fff79-gxwmp_da067996-34f2-4510-9dd1-4a5879f585ef/kube-rbac-proxy-web/0.log"
Jul 28 16:52:52.801279 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.801161    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555b4fff79-gxwmp_da067996-34f2-4510-9dd1-4a5879f585ef/kube-rbac-proxy/0.log"
Jul 28 16:52:52.833984 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.833951    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555b4fff79-gxwmp_da067996-34f2-4510-9dd1-4a5879f585ef/prom-label-proxy/0.log"
Jul 28 16:52:52.865221 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.865154    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555b4fff79-gxwmp_da067996-34f2-4510-9dd1-4a5879f585ef/kube-rbac-proxy-rules/0.log"
Jul 28 16:52:52.891798 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:52.891763    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555b4fff79-gxwmp_da067996-34f2-4510-9dd1-4a5879f585ef/kube-rbac-proxy-metrics/0.log"
Jul 28 16:52:55.276336 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.276303    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-l228b_68db1376-c928-43d3-af39-84d019ff6212/download-server/0.log"
Jul 28 16:52:55.322980 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.322932    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"]
Jul 28 16:52:55.323467 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.323443    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="gather"
Jul 28 16:52:55.323622 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.323609    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="gather"
Jul 28 16:52:55.323741 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.323729    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="copy"
Jul 28 16:52:55.323902 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.323887    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="copy"
Jul 28 16:52:55.324093 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.324079    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="copy"
Jul 28 16:52:55.324185 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.324175    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="013c910a-3d55-4b5b-91b5-622957bb6363" containerName="gather"
Jul 28 16:52:55.329639 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.329619    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.337127 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.337094    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"]
Jul 28 16:52:55.425809 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.425763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-lib-modules\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.425809 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.425803    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-proc\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.426066 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.425860    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4pt\" (UniqueName: \"kubernetes.io/projected/4b0424da-2459-44de-898b-459e3f2d3287-kube-api-access-6h4pt\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.426066 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.425901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-sys\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.426066 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.425939    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-podres\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526476 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-lib-modules\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526476 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-proc\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526476 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4pt\" (UniqueName: \"kubernetes.io/projected/4b0424da-2459-44de-898b-459e3f2d3287-kube-api-access-6h4pt\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-sys\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-podres\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526535    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-proc\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-lib-modules\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-podres\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.526755 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.526679    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0424da-2459-44de-898b-459e3f2d3287-sys\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.538045 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.538011    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4pt\" (UniqueName: \"kubernetes.io/projected/4b0424da-2459-44de-898b-459e3f2d3287-kube-api-access-6h4pt\") pod \"perf-node-gather-daemonset-xpfxr\" (UID: \"4b0424da-2459-44de-898b-459e3f2d3287\") " pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.643964 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.643709    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:55.805936 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:55.805850    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"]
Jul 28 16:52:55.811732 ip-10-0-133-120 kubenswrapper[2573]: W0728 16:52:55.811690    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b0424da_2459_44de_898b_459e3f2d3287.slice/crio-35f8bec2ed7c5600385ccf3592f315527b0d13cda800bb8a74dd6e5943dd0e00 WatchSource:0}: Error finding container 35f8bec2ed7c5600385ccf3592f315527b0d13cda800bb8a74dd6e5943dd0e00: Status 404 returned error can't find the container with id 35f8bec2ed7c5600385ccf3592f315527b0d13cda800bb8a74dd6e5943dd0e00
Jul 28 16:52:56.004083 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.004045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr" event={"ID":"4b0424da-2459-44de-898b-459e3f2d3287","Type":"ContainerStarted","Data":"72d5478294d32a9405c170e769b92d85b940754f4af7aaf7ec9c381651d59426"}
Jul 28 16:52:56.004266 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.004111    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr" event={"ID":"4b0424da-2459-44de-898b-459e3f2d3287","Type":"ContainerStarted","Data":"35f8bec2ed7c5600385ccf3592f315527b0d13cda800bb8a74dd6e5943dd0e00"}
Jul 28 16:52:56.004266 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.004144    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:52:56.022074 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.022011    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr" podStartSLOduration=1.021996091 podStartE2EDuration="1.021996091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:52:56.021097408 +0000 UTC m=+1961.414413946" watchObservedRunningTime="2026-07-28 16:52:56.021996091 +0000 UTC m=+1961.415312630"
Jul 28 16:52:56.746142 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.746097    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lp5c_ad54d500-f594-4094-8d9d-09e7c4a0f8fd/dns/0.log"
Jul 28 16:52:56.769748 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.769719    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4lp5c_ad54d500-f594-4094-8d9d-09e7c4a0f8fd/kube-rbac-proxy/0.log"
Jul 28 16:52:56.958006 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:56.957975    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wwffh_c1bca669-4acf-49c5-8686-5928990e6e9f/dns-node-resolver/0.log"
Jul 28 16:52:57.490193 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:57.490161    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vm6qz_b7a9e889-bb8a-4c9d-8fe0-12f2200c179c/node-ca/0.log"
Jul 28 16:52:58.624605 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:58.624568    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qzdmk_6e6b08cf-1d43-4000-9fb3-592b2bbd5e6f/serve-healthcheck-canary/0.log"
Jul 28 16:52:59.165515 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:59.165488    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bfj8q_7599f6b8-1ce8-4340-973e-367c1b8c778d/kube-rbac-proxy/0.log"
Jul 28 16:52:59.190166 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:59.190143    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bfj8q_7599f6b8-1ce8-4340-973e-367c1b8c778d/exporter/0.log"
Jul 28 16:52:59.213360 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:52:59.213335    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bfj8q_7599f6b8-1ce8-4340-973e-367c1b8c778d/extractor/0.log"
Jul 28 16:53:01.399981 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:01.399952    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-c6rqv_87c02573-14b6-48ff-8e3a-e725561fcefe/server/0.log"
Jul 28 16:53:01.611740 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:01.611710    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-s7ln6_c1e68a70-7e21-449f-b40d-0117cff07ee6/manager/0.log"
Jul 28 16:53:01.654356 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:01.654282    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-tc5sd_be88d655-6335-4d8d-b1cb-9ae7c96929cc/s3-init/0.log"
Jul 28 16:53:01.674237 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:01.674216    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-tc5sd_be88d655-6335-4d8d-b1cb-9ae7c96929cc/download-hf-model/0.log"
Jul 28 16:53:01.707388 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:01.707356    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-kt8h5_6cb7d36e-8fca-4fce-ba27-31e8e4c57b4a/seaweedfs/0.log"
Jul 28 16:53:02.021709 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:02.021685    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kvj4s/perf-node-gather-daemonset-xpfxr"
Jul 28 16:53:07.206706 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.206641    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:53:07.237420 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.237397    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/egress-router-binary-copy/0.log"
Jul 28 16:53:07.269560 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.269537    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/cni-plugins/0.log"
Jul 28 16:53:07.297744 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.297709    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/bond-cni-plugin/0.log"
Jul 28 16:53:07.323557 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.323534    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/routeoverride-cni/0.log"
Jul 28 16:53:07.353526 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.353506    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/whereabouts-cni-bincopy/0.log"
Jul 28 16:53:07.392118 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.392088    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hfsv_ff3b52ec-8aa9-44bc-aa2f-14eb3a998ba9/whereabouts-cni/0.log"
Jul 28 16:53:07.953950 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:07.953921    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mf4gt_34b01b86-43a7-4891-aa95-8bd3ce6a0bf6/kube-multus/0.log"
Jul 28 16:53:08.188512 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:08.188479    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nfzgs_6ee98b90-1ba1-4013-bf30-8b89f1a028a6/network-metrics-daemon/0.log"
Jul 28 16:53:08.236905 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:08.236886    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nfzgs_6ee98b90-1ba1-4013-bf30-8b89f1a028a6/kube-rbac-proxy/0.log"
Jul 28 16:53:09.561899 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.561868    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-controller/0.log"
Jul 28 16:53:09.604625 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.604601    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/0.log"
Jul 28 16:53:09.604779 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.604717    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovn-acl-logging/1.log"
Jul 28 16:53:09.639066 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.639046    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/kube-rbac-proxy-node/0.log"
Jul 28 16:53:09.675974 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.675934    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 16:53:09.717431 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.717410    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/northd/0.log"
Jul 28 16:53:09.764223 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.764206    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/nbdb/0.log"
Jul 28 16:53:09.807156 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.807138    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/sbdb/0.log"
Jul 28 16:53:09.882498 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:09.882440    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pszj4_0c1f4c4a-a7f8-4994-a65c-2ffe3aa0ad97/ovnkube-controller/0.log"
Jul 28 16:53:10.895299 ip-10-0-133-120 kubenswrapper[2573]: I0728 16:53:10.895267    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-6h5n9_e2e6c0eb-781d-4bf5-92e0-6a4de60d9b57/network-check-target-container/0.log"