Jul 28 12:32:59.700183 ip-10-0-129-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 12:32:59.700193 ip-10-0-129-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 12:32:59.700202 ip-10-0-129-190 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 12:32:59.700540 ip-10-0-129-190 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 12:33:09.760001 ip-10-0-129-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 12:33:09.760017 ip-10-0-129-190 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d638dad0b7f14fdeb2b15a54e20dbf2b --
Jul 28 12:35:06.973681 ip-10-0-129-190 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:35:07.402476 ip-10-0-129-190 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:35:07.402476 ip-10-0-129-190 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:35:07.402476 ip-10-0-129-190 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:35:07.402476 ip-10-0-129-190 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:35:07.402476 ip-10-0-129-190 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:35:07.403603 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.403377    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:35:07.408261 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408239    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:07.408261 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408256    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:07.408261 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408262    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:07.408261 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408267    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:07.408261 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408271    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408278    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408283    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408287    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408291    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408295    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408299    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408303    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408307    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408311    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408315    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408320    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408324    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408328    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408332    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408336    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408342    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408357    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408361    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408366    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:07.408615 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408370    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408374    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408379    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408383    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408387    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408391    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408397    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408401    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408406    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408410    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408415    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408420    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408425    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408430    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408435    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408440    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408463    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408468    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408474    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408478    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:07.409580 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408483    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408488    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408493    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408508    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408513    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408521    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408526    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408531    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408537    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408542    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408547    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408552    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408556    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408561    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408565    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408569    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408574    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408578    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408583    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408588    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:07.410429 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408593    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408597    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408602    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408607    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408611    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408615    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408620    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408627    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408631    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408636    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408641    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408646    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408654    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408660    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408665    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408670    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408675    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408680    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408684    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:07.411000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408689    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408694    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408699    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408706    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408711    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408716    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408720    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408725    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408729    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408734    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408738    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408743    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408748    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408753    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408757    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.408762    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410086    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410098    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410104    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:07.411811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410110    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410115    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410122    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410127    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410132    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410137    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410142    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410147    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410152    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410156    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410161    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410165    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410169    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410174    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410179    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410184    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410188    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410192    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410197    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410201    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:07.412687 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410206    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410210    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410215    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410219    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410224    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410228    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410233    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410239    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410244    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410248    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410252    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410258    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410263    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410268    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410273    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410279    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410283    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410288    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410292    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:07.413657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410297    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.410301    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411141    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411153    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411159    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411167    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411174    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411179    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411184    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411189    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411195    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411200    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411205    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411209    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411215    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411219    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411226    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411233    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411237    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:07.414486 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411242    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411247    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411252    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411257    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411262    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411267    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411272    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411277    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411282    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411286    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411293    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411298    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411307    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411313    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411317    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411322    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411328    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411333    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411338    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411343    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:07.415051 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411348    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411353    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411358    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411363    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411367    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411371    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411376    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411381    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411386    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411391    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411395    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411400    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411404    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411409    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411413    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411418    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411423    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.411429    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412083    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412092    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:07.415638 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412097    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412102    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412108    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412113    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412118    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412124    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412128    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412132    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412137    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412142    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412146    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412151    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412156    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412160    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412165    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412169    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412174    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412179    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412183    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:07.416164 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412188    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412192    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412197    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412201    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412206    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412211    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412216    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412220    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412225    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412229    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412235    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412240    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412244    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412249    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412253    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412258    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412263    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412268    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412272    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412276    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:07.416742 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412284    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412289    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412294    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412298    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412303    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412308    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412312    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412317    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412324    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412328    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412333    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412338    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412342    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412347    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412351    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412356    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412361    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412366    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412370    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412375    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:07.417269 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412380    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412385    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412390    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412395    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412399    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412404    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412409    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412414    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412419    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412424    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412429    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412433    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412438    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412462    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412467    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412472    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412476    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412481    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412485    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412489    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:07.417817 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412496    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412502    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412507    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412512    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412516    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412521    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412525    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412530    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412538    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412544    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412550    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412554    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412559    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412564    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412569    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412574    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412579    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.412584    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412700    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412712    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:35:07.418371 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412725    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412732    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412740    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412746    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412754    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412761    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412766    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412772    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412778    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412785    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412790    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412796    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412801    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412806    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412812    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412817    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412824    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412829    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412834    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412840    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412846    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412852    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412857    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:35:07.418941 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412863    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412868    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412874    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412879    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412885    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412890    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412895    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412902    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412907    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412913    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412918    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412924    2576 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412930    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412938    2576 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412943    2576 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412949    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412954    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412960    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412966    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412971    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412976    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412985    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412990    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.412995    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413000    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:35:07.419606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413006    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413011    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413016    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413022    2576 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413028    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413033    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413039    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413045    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413050    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413057    2576 flags.go:64] FLAG: --help="false"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413062    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413067    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413072    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413078    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413084    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413090    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413095    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413100    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413105    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413110    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413117    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413123    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413128    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413133    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:35:07.420264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413138    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413144    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413149    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413154    2576 flags.go:64] FLAG: --lock-file=""
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413160    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413166    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413171    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413180    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413185    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413191    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413196    2576 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413202    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413208    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413213    2576 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413218    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413229    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413235    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413242    2576 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413248    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413253    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413258    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413263    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413268    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413274    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413279    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:35:07.420957 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413292    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413298    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413303    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413308    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413314    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413323    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413328    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413334    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413339    2576 flags.go:64] FLAG: --port="10250"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413344    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413349    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0936649e5e88f9233"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413355    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413362    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413368    2576 flags.go:64] FLAG: --register-node="true"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413374    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413380    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413386    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413391    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413396    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413403    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413408    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413414    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413419    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413424    2576 flags.go:64] FLAG: --runonce="false"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413429    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:35:07.421634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413435    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413440    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413471    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413476    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413482    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413487    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413493    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413498    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413503    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413508    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413513    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413518    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413525    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413530    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413540    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413545    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413551    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413557    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413562    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413569    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413574    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413582    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413587    2576 flags.go:64] FLAG: --v="2"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413595    2576 flags.go:64] FLAG: --version="false"
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413602    2576 flags.go:64] FLAG: --vmodule=""
Jul 28 12:35:07.422352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413609    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.413615    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413811    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413820    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413825    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413830    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413835    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413840    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413844    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413849    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413854    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413858    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413863    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413867    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413872    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413876    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413881    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413886    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413891    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413896    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:07.423039 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413903    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413908    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413912    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413917    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413921    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413926    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413933    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413937    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413943    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413948    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413953    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413957    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413961    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413966    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413971    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413976    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413980    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413985    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413990    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:07.423592 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.413997    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414003    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414009    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414013    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414020    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414025    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414030    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414034    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414039    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414043    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414048    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414052    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414057    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414062    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414067    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414072    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414077    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414082    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414086    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:07.424096 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414093    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414100    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414109    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414115    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414123    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414128    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414134    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414138    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414143    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414148    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414152    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414157    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414162    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414166    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414171    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414176    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414181    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414185    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414190    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:07.424885 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414194    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414199    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414204    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414208    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414213    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414217    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414222    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414227    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414232    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414237    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414241    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414246    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414251    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414257    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414262    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414268    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414272    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414277    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414281    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:07.425625 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414286    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414290    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414295    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414300    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.414304    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.414947    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.422297    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.422313    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.422328    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.422335    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422404    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422409    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422413    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422417    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422420    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:07.426219 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422424    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422427    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422431    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422434    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422437    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422440    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422457    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422460    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422464    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422467    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422470    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422473    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422476    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422479    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422481    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422484    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422488    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422491    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422494    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422497    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:07.426701 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422500    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422503    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422506    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422508    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422514    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422517    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422520    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422525    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422529    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422532    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422535    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422538    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422541    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422544    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422547    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422550    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422553    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422556    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422558    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422561    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:07.427223 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422564    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422567    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422569    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422572    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422575    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422578    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422580    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422583    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422586    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422589    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422592    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422596    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422600    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422605    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422609    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422613    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422617    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422621    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422624    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:07.427781 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422628    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422630    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422634    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422636    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422639    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422642    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422645    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422647    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422650    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422653    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422656    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422658    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422661    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422664    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422667    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422670    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422672    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422675    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422678    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422681    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:07.428268 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422683    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422686    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422689    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422692    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422695    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422698    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422701    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422704    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422707    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422711    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422714    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422717    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422720    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422723    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422726    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:07.428807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.422731    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422873    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422879    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422882    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422885    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422888    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422891    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422894    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422897    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422900    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422902    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422905    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422908    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422911    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422914    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422916    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422919    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422922    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422925    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422928    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:35:07.429222 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422931    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422935    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422939    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422942    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422945    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422948    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422951    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422954    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422957    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422962    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422965    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422967    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422970    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422973    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422976    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422979    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422982    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422984    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422987    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:35:07.429747 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422990    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422993    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422996    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.422998    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423001    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423004    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423007    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423010    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423013    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423016    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423018    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423021    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423024    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423026    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423029    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423032    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423035    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423038    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423041    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423044    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:35:07.430242 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423047    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423050    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423053    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423057    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423061    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423064    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423067    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423070    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423073    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423075    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423078    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423081    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423083    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423086    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423089    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423091    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423094    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423098    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423102    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:35:07.430894 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423105    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423108    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423112    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423115    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423118    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423120    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423123    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423126    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423129    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423131    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423134    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423137    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423139    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423142    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423145    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423147    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423150    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423153    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423156    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423159    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:35:07.431399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423161    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:07.423164    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.423169    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.423287    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.427327    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.428996    2576 server.go:1019] "Starting client certificate rotation"
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.429099    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.429142    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:35:07.431963 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.431730    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:35:07.433016 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.432997    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:35:07.441195 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.441176    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:35:07.446455 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.446428    2576 log.go:25] "Validated CRI v1 image API"
Jul 28 12:35:07.447261 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.447244    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:35:07.450121 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.450102    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d26a34e0-db0e-4690-9997-3a3bbbc91ace:/dev/nvme0n1p4 d79886db-49ee-4f05-8bb8-4d434509510a:/dev/nvme0n1p3]
Jul 28 12:35:07.450177 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.450122    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:35:07.456730 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.456611    2576 manager.go:216] Machine: {Timestamp:2026-07-28 12:35:07.455058293 +0000 UTC m=+0.356202677 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3130133 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d3d15dabef2ceb1ca49015bcd31c SystemUUID:ec28d3d1-5dab-ef2c-eb1c-a49015bcd31c BootID:d638dad0-b7f1-4fde-b2b1-5a54e20dbf2b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:cb:44:00:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:cb:44:00:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:ec:87:a7:78:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:35:07.456730 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.456727    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:35:07.456854 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.456826    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:35:07.457379 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.457365    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:35:07.460068 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.460038    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:35:07.460224 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.460072    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:35:07.461218 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.461207    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:35:07.461260 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.461221    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:35:07.461260 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.461229    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:35:07.461978 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.461967    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:35:07.463152 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.463141    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:35:07.467093 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467069    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:35:07.467235 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467224    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:35:07.467265 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467259    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:35:07.467301 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467285    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:35:07.467335 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467308    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:35:07.467335 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467323    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:35:07.467398 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467336    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:35:07.467856 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467833    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:35:07.467952 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.467863    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:35:07.470611 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.470583    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:35:07.472172 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472147    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:35:07.472267 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472197    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:35:07.472904 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472891    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:35:07.472953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472910    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:35:07.472953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472919    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:35:07.472953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472928    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:35:07.472953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472949    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472956    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472965    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472971    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472978    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472984    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.472992    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:35:07.473067 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.473000    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:35:07.473846 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.473835    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:35:07.473885 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.473848    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:35:07.477332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.477317    2576 server.go:1268] "Started kubelet"
Jul 28 12:35:07.477500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.477448    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:35:07.477615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.477599    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:35:07.477708 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.477680    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:35:07.478412 ip-10-0-129-190 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:35:07.479113 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.479044    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:35:07.479113 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.479042    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:35:07.479244 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.479201    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:35:07.479531 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.479407    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:35:07.480407 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.480372    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:35:07.484070 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.484051    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:35:07.485087 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.483577    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18c6743cf640ed04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-07-28 12:35:07.477290244 +0000 UTC m=+0.378434627,LastTimestamp:2026-07-28 12:35:07.477290244 +0000 UTC m=+0.378434627,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Jul 28 12:35:07.485820 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.485803    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:35:07.485891 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.485810    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:35:07.485944 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.485930    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.486505 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486487    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:35:07.486617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486602    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:35:07.486684 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486621    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:35:07.486773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486752    2576 factory.go:55] Registering systemd factory
Jul 28 12:35:07.486773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486766    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:35:07.486773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486772    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:35:07.486773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.486776    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:35:07.487036 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.487021    2576 factory.go:153] Registering CRI-O factory
Jul 28 12:35:07.487096 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.487042    2576 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:35:07.487164 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.487094    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:35:07.487164 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.487121    2576 factory.go:103] Registering Raw factory
Jul 28 12:35:07.487273 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.487177    2576 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:35:07.487629 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.487613    2576 manager.go:349] Starting recovery of all containers
Jul 28 12:35:07.488040 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.488016    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgzcp"
Jul 28 12:35:07.494849 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.494818    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:35:07.494988 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.494964    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:35:07.496856 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.496817    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hgzcp"
Jul 28 12:35:07.499235 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.499217    2576 manager.go:354] Recovery completed
Jul 28 12:35:07.504797 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.504781    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.507469 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.507438    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.507538 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.507484    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.507538 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.507496    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.508017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.507999    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:35:07.508017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.508007    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.508017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.508013    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:35:07.508194 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.508030    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:35:07.508817 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.508801    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.508874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.508827    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.508874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.508839    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.509203 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.509110    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18c6743cf80d6431  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-07-28 12:35:07.507467313 +0000 UTC m=+0.408611696,LastTimestamp:2026-07-28 12:35:07.507467313 +0000 UTC m=+0.408611696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Jul 28 12:35:07.510137 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.510124    2576 policy_none.go:49] "None policy: Start"
Jul 28 12:35:07.510191 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.510140    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:35:07.510191 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.510153    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:35:07.512266 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.512255    2576 policy_none.go:47] "Start"
Jul 28 12:35:07.552283 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552095    2576 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:35:07.552403 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.552324    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:35:07.552403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552337    2576 server.go:87] "Starting device plugin registration server"
Jul 28 12:35:07.552642 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552628    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:35:07.552711 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552645    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:35:07.552763 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552752    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:35:07.552871 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552854    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:35:07.552871 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.552868    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:35:07.554324 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.553698    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:35:07.554324 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.553760    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.654250 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.654184    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.656497 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.655831    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.656497 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.655864    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.656497 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.655875    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.656497 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.655904    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.664500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.664484    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.664550 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.664508    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-190.ec2.internal\": node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.680309 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.680287    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.712870 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.712831    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:35:07.714301 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.714279    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:35:07.714424 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.714321    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:35:07.714424 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.714346    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:35:07.714424 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.714382    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:35:07.716896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.716878    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:07.782900 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.782849    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.815791 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.815758    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"]
Jul 28 12:35:07.815878 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.815848    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.817757 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.817737    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.817850 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.817772    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.817850 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.817790    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.820304 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.820286    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.820495 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.820478    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.820573 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.820527    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.821193 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.821177    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.821263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.821205    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.821263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.821219    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.821554 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.821540    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.821599 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.821560    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.821599 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.821574    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.824559 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.824541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.824650 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.824577    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:35:07.825815 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.825789    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:35:07.825815 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.825819    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:35:07.825983 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.825830    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:35:07.846116 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.846088    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-190.ec2.internal\" not found" node="ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.850801 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.850783    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-190.ec2.internal\" not found" node="ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.883269 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.883246    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.890090 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.890069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d428843330d938518f1beb905f7443e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"d428843330d938518f1beb905f7443e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.890156 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.890097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d428843330d938518f1beb905f7443e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"d428843330d938518f1beb905f7443e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.890156 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.890129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1705df2f09f7b34e26c5e6a8a8046a32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-190.ec2.internal\" (UID: \"1705df2f09f7b34e26c5e6a8a8046a32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.984356 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:07.984275    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:07.991141 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.991121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d428843330d938518f1beb905f7443e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"d428843330d938518f1beb905f7443e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.991203 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.991150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d428843330d938518f1beb905f7443e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"d428843330d938518f1beb905f7443e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.991203 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.991168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1705df2f09f7b34e26c5e6a8a8046a32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-190.ec2.internal\" (UID: \"1705df2f09f7b34e26c5e6a8a8046a32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.991270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.991203    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1705df2f09f7b34e26c5e6a8a8046a32-config\") pod \"kube-apiserver-proxy-ip-10-0-129-190.ec2.internal\" (UID: \"1705df2f09f7b34e26c5e6a8a8046a32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.991270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.991232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d428843330d938518f1beb905f7443e6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"d428843330d938518f1beb905f7443e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:07.991270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:07.991237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d428843330d938518f1beb905f7443e6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"d428843330d938518f1beb905f7443e6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:08.092030 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:08.091976    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:08.152857 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.152824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:08.154017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.153990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:08.193117 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:08.193086    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:08.294226 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:08.294136    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:08.394486 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:08.394451    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Jul 28 12:35:08.428671 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.428652    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:35:08.429114 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.428774    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:35:08.429419 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.429385    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:08.475800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.475774    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:08.486512 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.486480    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:35:08.486670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.486490    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:08.499486 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.499463    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:35:08.500678 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.500664    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Jul 28 12:35:08.502227 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.502211    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:35:08.504482 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.504457    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:30:08 +0000 UTC" deadline="2027-12-28 12:10:59.672207817 +0000 UTC"
Jul 28 12:35:08.504525 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.504483    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12431h35m51.16772741s"
Jul 28 12:35:08.510622 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.510605    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:35:08.534841 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.534817    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5nmtp"
Jul 28 12:35:08.543313 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.543294    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5nmtp"
Jul 28 12:35:08.864120 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:08.864066    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd428843330d938518f1beb905f7443e6.slice/crio-d3d50cfa742533a6bac29c8bdbb10015ce45d02cda914729df155bd2de66236a WatchSource:0}: Error finding container d3d50cfa742533a6bac29c8bdbb10015ce45d02cda914729df155bd2de66236a: Status 404 returned error can't find the container with id d3d50cfa742533a6bac29c8bdbb10015ce45d02cda914729df155bd2de66236a
Jul 28 12:35:08.864362 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:08.864338    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1705df2f09f7b34e26c5e6a8a8046a32.slice/crio-fae947332333633bce99a9070a0dc2c24ab705898ae38d8923d3bd68dcae0552 WatchSource:0}: Error finding container fae947332333633bce99a9070a0dc2c24ab705898ae38d8923d3bd68dcae0552: Status 404 returned error can't find the container with id fae947332333633bce99a9070a0dc2c24ab705898ae38d8923d3bd68dcae0552
Jul 28 12:35:08.868370 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.868354    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:35:08.992226 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:08.992138    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:09.470611 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.470569    2576 apiserver.go:52] "Watching apiserver"
Jul 28 12:35:09.482464 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.482412    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:35:09.483093 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.483023    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7","openshift-dns/node-resolver-js4c9","openshift-image-registry/node-ca-kszkc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal","openshift-multus/multus-additional-cni-plugins-zdksc","openshift-multus/multus-ftf2m","openshift-multus/network-metrics-daemon-hcvxx","kube-system/global-pull-secret-syncer-rg6kq","kube-system/konnectivity-agent-hmz9l","openshift-cluster-node-tuning-operator/tuned-nprcv","openshift-network-diagnostics/network-check-target-h9c6h","openshift-network-operator/iptables-alerter-vqr7r","openshift-ovn-kubernetes/ovnkube-node-x9s78"]
Jul 28 12:35:09.485953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.485925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.489865 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.489068    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.491557 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.491418    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.491675 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.491652    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:35:09.491799 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.491733    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.491888 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.491869    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bddgz\""
Jul 28 12:35:09.492027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.491902    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.492379 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.492358    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fxt5v\""
Jul 28 12:35:09.492503 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.492365    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.493201 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.492794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.495587 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.495498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.496338 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.495893    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:35:09.496769 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.496471    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.496769 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.496662    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.496769 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.496734    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tz4zx\""
Jul 28 12:35:09.499903 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.499551    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:35:09.499903 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.499630    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.499903 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.499766    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.500185 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:35:09.500252 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500213    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p7l7z\""
Jul 28 12:35:09.500515 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-registration-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500599 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-device-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500599 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9942989-507a-4a45-a294-217549de52ce-hosts-file\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.500599 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9942989-507a-4a45-a294-217549de52ce-tmp-dir\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.500773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-socket-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-etc-selinux\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-sys-fs\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcqk\" (UniqueName: \"kubernetes.io/projected/7b84db21-8801-4bbd-b7a1-cdf05b764156-kube-api-access-tqcqk\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.500773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4mc\" (UniqueName: \"kubernetes.io/projected/f9942989-507a-4a45-a294-217549de52ce-kube-api-access-vx4mc\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.501088 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.500937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.503286 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.503241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:35:09.504000 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.503978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:09.504090 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.504069    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:09.504624 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.504572    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:35:09.505009 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.504961    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f5djj\""
Jul 28 12:35:09.506034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.505936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.506034 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.506005    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:09.508734 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.508710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.511090 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.511062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.511616 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.511590    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:35:09.512975 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.512952    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:35:09.513580 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.513553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:09.513697 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.513647    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:09.514237 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.514214    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xwhmc\""
Jul 28 12:35:09.515029 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.515006    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.515749 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.515725    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.516370 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.516072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.519483 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.518858    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8hrfk\""
Jul 28 12:35:09.519483 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.519424    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.521550 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.521527    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.521647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.521623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5kjk\""
Jul 28 12:35:09.522466 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.522437    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:35:09.523220 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.523199    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.524717 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.524685    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:35:09.524990 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.524970    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:35:09.525208 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.525191    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:35:09.526518 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.526495    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:35:09.526621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.526527    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:35:09.526802 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.526781    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-frd7f\""
Jul 28 12:35:09.527031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.527013    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:35:09.546197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.546150    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:30:08 +0000 UTC" deadline="2028-03-08 19:05:24.419215324 +0000 UTC"
Jul 28 12:35:09.546197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.546196    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14142h30m14.873023917s"
Jul 28 12:35:09.587882 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.587858    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:35:09.601914 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.601879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-etc-selinux\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.601927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcqk\" (UniqueName: \"kubernetes.io/projected/7b84db21-8801-4bbd-b7a1-cdf05b764156-kube-api-access-tqcqk\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.601961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-k8s-cni-cncf-io\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.601993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysconfig\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-kubernetes\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-cnibin\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.602094 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-log-socket\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovn-node-metrics-cert\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95bada68-c40f-4006-a489-e937d2475a6d-iptables-alerter-script\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b54e8191-d84d-430d-9bb1-1c5f76787bfe-host\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5352b7da-d4d1-4cca-9d6e-d522d0495f42-kubelet-config\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysctl-d\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-systemd\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-host\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-system-cni-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-etc-selinux\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.602487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-conf-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-daemon-config\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-cni-bin\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-cni-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-system-cni-dir\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5352b7da-d4d1-4cca-9d6e-d522d0495f42-dbus\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.602964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-device-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-device-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9942989-507a-4a45-a294-217549de52ce-hosts-file\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-systemd\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-etc-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603166    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9942989-507a-4a45-a294-217549de52ce-hosts-file\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdj9c\" (UniqueName: \"kubernetes.io/projected/fd5179df-c0f5-4db1-a387-021f97d1cbd8-kube-api-access-vdj9c\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-netns\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-kubelet\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-env-overrides\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6538151-c128-4020-98b4-0e9bb1959506-tmp\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-cni-binary-copy\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95bada68-c40f-4006-a489-e937d2475a6d-host-slash\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9942989-507a-4a45-a294-217549de52ce-tmp-dir\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-node-log\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovnkube-config\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgz8\" (UniqueName: \"kubernetes.io/projected/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-kube-api-access-7jgz8\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-cni-netd\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-sys-fs\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-systemd-units\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-slash\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.603783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6342dafc-bb6e-4bda-9dcd-9afed70b8a6a-konnectivity-ca\") pod \"konnectivity-agent-hmz9l\" (UID: \"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a\") " pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysctl-conf\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jr6\" (UniqueName: \"kubernetes.io/projected/b54e8191-d84d-430d-9bb1-1c5f76787bfe-kube-api-access-g8jr6\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-hostroot\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-var-lib-kubelet\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnsd\" (UniqueName: \"kubernetes.io/projected/f6538151-c128-4020-98b4-0e9bb1959506-kube-api-access-gwnsd\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603892    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9942989-507a-4a45-a294-217549de52ce-tmp-dir\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-run-netns\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-ovn\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.603998    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-cnibin\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604059    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-sys-fs\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6cq\" (UniqueName: \"kubernetes.io/projected/95bada68-c40f-4006-a489-e937d2475a6d-kube-api-access-db6cq\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b54e8191-d84d-430d-9bb1-1c5f76787bfe-serviceca\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-cni-bin\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.604641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-modprobe-d\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-sys\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-socket-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovnkube-script-lib\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-etc-kubernetes\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-socket-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-lib-modules\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-os-release\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-registration-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-os-release\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a92a412-e8e8-4181-8d3e-d2388aea297d-cni-binary-copy\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6342dafc-bb6e-4bda-9dcd-9afed70b8a6a-agent-certs\") pod \"konnectivity-agent-hmz9l\" (UID: \"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a\") " pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-kubelet\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.605670 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4mc\" (UniqueName: \"kubernetes.io/projected/f9942989-507a-4a45-a294-217549de52ce-kube-api-access-vx4mc\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-socket-dir-parent\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-cni-multus\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-multus-certs\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6538151-c128-4020-98b4-0e9bb1959506-etc-tuned\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b84db21-8801-4bbd-b7a1-cdf05b764156-registration-dir\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.604968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-var-lib-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.605003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgb8h\" (UniqueName: \"kubernetes.io/projected/0a92a412-e8e8-4181-8d3e-d2388aea297d-kube-api-access-bgb8h\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.605030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-run\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.606460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.605080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4k8m\" (UniqueName: \"kubernetes.io/projected/01ed658a-d837-4ae6-a286-222f1cda2c28-kube-api-access-b4k8m\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.609541 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.609503    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:35:09.615835 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.615808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4mc\" (UniqueName: \"kubernetes.io/projected/f9942989-507a-4a45-a294-217549de52ce-kube-api-access-vx4mc\") pod \"node-resolver-js4c9\" (UID: \"f9942989-507a-4a45-a294-217549de52ce\") " pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.615959 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.615819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcqk\" (UniqueName: \"kubernetes.io/projected/7b84db21-8801-4bbd-b7a1-cdf05b764156-kube-api-access-tqcqk\") pod \"aws-ebs-csi-driver-node-4t7k7\" (UID: \"7b84db21-8801-4bbd-b7a1-cdf05b764156\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.705800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-kubelet\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-socket-dir-parent\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-kubelet\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-cni-multus\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-cni-multus\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-multus-certs\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6538151-c128-4020-98b4-0e9bb1959506-etc-tuned\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-var-lib-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.705980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgb8h\" (UniqueName: \"kubernetes.io/projected/0a92a412-e8e8-4181-8d3e-d2388aea297d-kube-api-access-bgb8h\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.705999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-run\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706017    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-socket-dir-parent\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4k8m\" (UniqueName: \"kubernetes.io/projected/01ed658a-d837-4ae6-a286-222f1cda2c28-kube-api-access-b4k8m\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-k8s-cni-cncf-io\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-var-lib-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706108    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysconfig\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-multus-certs\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-kubernetes\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706162    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-cnibin\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-log-socket\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovn-node-metrics-cert\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95bada68-c40f-4006-a489-e937d2475a6d-iptables-alerter-script\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b54e8191-d84d-430d-9bb1-1c5f76787bfe-host\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.706500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5352b7da-d4d1-4cca-9d6e-d522d0495f42-kubelet-config\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysctl-d\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-systemd\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-host\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-system-cni-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-conf-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-daemon-config\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-cni-bin\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-cni-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-system-cni-dir\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5352b7da-d4d1-4cca-9d6e-d522d0495f42-dbus\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-systemd\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-etc-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdj9c\" (UniqueName: \"kubernetes.io/projected/fd5179df-c0f5-4db1-a387-021f97d1cbd8-kube-api-access-vdj9c\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-netns\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.707197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-kubelet\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.707916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.706864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-env-overrides\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.707916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.707499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6538151-c128-4020-98b4-0e9bb1959506-tmp\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.707916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.707570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-cni-binary-copy\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.710206 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-run\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.710344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95bada68-c40f-4006-a489-e937d2475a6d-host-slash\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.710344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-node-log\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710298    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysconfig\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.710344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovnkube-config\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710576 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-node-log\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710576 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgz8\" (UniqueName: \"kubernetes.io/projected/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-kube-api-access-7jgz8\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:09.710576 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-cni-netd\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710576 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-systemd-units\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710576 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-slash\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710576 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6342dafc-bb6e-4bda-9dcd-9afed70b8a6a-konnectivity-ca\") pod \"konnectivity-agent-hmz9l\" (UID: \"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a\") " pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysctl-conf\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jr6\" (UniqueName: \"kubernetes.io/projected/b54e8191-d84d-430d-9bb1-1c5f76787bfe-kube-api-access-g8jr6\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-hostroot\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95bada68-c40f-4006-a489-e937d2475a6d-host-slash\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-var-lib-kubelet\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnsd\" (UniqueName: \"kubernetes.io/projected/f6538151-c128-4020-98b4-0e9bb1959506-kube-api-access-gwnsd\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-var-lib-kubelet\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-run-netns\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-ovn\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.710892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-k8s-cni-cncf-io\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.711488 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.710914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-run-netns\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.711488 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.711015    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:09.711488 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.711043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-cni-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.711488 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.711148    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:10.211088498 +0000 UTC m=+3.112232893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.711801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-hostroot\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovnkube-config\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.711995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysctl-conf\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-cnibin\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6cq\" (UniqueName: \"kubernetes.io/projected/95bada68-c40f-4006-a489-e937d2475a6d-kube-api-access-db6cq\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6342dafc-bb6e-4bda-9dcd-9afed70b8a6a-konnectivity-ca\") pod \"konnectivity-agent-hmz9l\" (UID: \"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a\") " pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b54e8191-d84d-430d-9bb1-1c5f76787bfe-serviceca\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-cni-bin\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712374    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-cni-netd\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-modprobe-d\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-kubernetes\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712512    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-cnibin\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-ovn\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-systemd-units\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712692    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-log-socket\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-sys\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.713031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-systemd\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712875    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-cni-bin\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.712955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-modprobe-d\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-slash\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-conf-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-cnibin\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713491    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-host\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713526    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-system-cni-dir\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713598    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-run-netns\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713670    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-host-var-lib-kubelet\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovnkube-script-lib\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713761    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b54e8191-d84d-430d-9bb1-1c5f76787bfe-host\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-system-cni-dir\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.714073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-etc-kubernetes\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-lib-modules\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.713978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-lib-modules\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-etc-kubernetes\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714163    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-etc-sysctl-d\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714230    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5352b7da-d4d1-4cca-9d6e-d522d0495f42-kubelet-config\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-env-overrides\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714288    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-etc-openvswitch\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6538151-c128-4020-98b4-0e9bb1959506-sys\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-run-systemd\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovnkube-script-lib\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714546    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5352b7da-d4d1-4cca-9d6e-d522d0495f42-dbus\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-os-release\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.715001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:09.715896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.715411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/01ed658a-d837-4ae6-a286-222f1cda2c28-os-release\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.715896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.714744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-os-release\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.715896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.715752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd5179df-c0f5-4db1-a387-021f97d1cbd8-host-cni-bin\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.716353 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.716332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a92a412-e8e8-4181-8d3e-d2388aea297d-multus-daemon-config\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.716353 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.716337    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.716498 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.716401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a92a412-e8e8-4181-8d3e-d2388aea297d-cni-binary-copy\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.716498 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.716435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6342dafc-bb6e-4bda-9dcd-9afed70b8a6a-agent-certs\") pod \"konnectivity-agent-hmz9l\" (UID: \"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a\") " pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.716637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0a92a412-e8e8-4181-8d3e-d2388aea297d-os-release\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.716653    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.716739    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:10.216715685 +0000 UTC m=+3.117860058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.717162    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a92a412-e8e8-4181-8d3e-d2388aea297d-cni-binary-copy\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.717264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6538151-c128-4020-98b4-0e9bb1959506-etc-tuned\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.717655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b54e8191-d84d-430d-9bb1-1c5f76787bfe-serviceca\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.718194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95bada68-c40f-4006-a489-e937d2475a6d-iptables-alerter-script\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.718615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.718502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01ed658a-d837-4ae6-a286-222f1cda2c28-cni-binary-copy\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.720219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.720157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6538151-c128-4020-98b4-0e9bb1959506-tmp\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.720693 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.720585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgb8h\" (UniqueName: \"kubernetes.io/projected/0a92a412-e8e8-4181-8d3e-d2388aea297d-kube-api-access-bgb8h\") pod \"multus-ftf2m\" (UID: \"0a92a412-e8e8-4181-8d3e-d2388aea297d\") " pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.721987 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.721806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd5179df-c0f5-4db1-a387-021f97d1cbd8-ovn-node-metrics-cert\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.724910 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.724850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"d428843330d938518f1beb905f7443e6","Type":"ContainerStarted","Data":"d3d50cfa742533a6bac29c8bdbb10015ce45d02cda914729df155bd2de66236a"}
Jul 28 12:35:09.725060 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.725034    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6342dafc-bb6e-4bda-9dcd-9afed70b8a6a-agent-certs\") pod \"konnectivity-agent-hmz9l\" (UID: \"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a\") " pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.725251 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.725232    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:09.725349 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.725257    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:09.725349 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.725273    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:09.725349 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:09.725347    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:10.2253296 +0000 UTC m=+3.126473972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:09.730017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.729955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal" event={"ID":"1705df2f09f7b34e26c5e6a8a8046a32","Type":"ContainerStarted","Data":"fae947332333633bce99a9070a0dc2c24ab705898ae38d8923d3bd68dcae0552"}
Jul 28 12:35:09.731171 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.731150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgz8\" (UniqueName: \"kubernetes.io/projected/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-kube-api-access-7jgz8\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:09.731795 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.731757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jr6\" (UniqueName: \"kubernetes.io/projected/b54e8191-d84d-430d-9bb1-1c5f76787bfe-kube-api-access-g8jr6\") pod \"node-ca-kszkc\" (UID: \"b54e8191-d84d-430d-9bb1-1c5f76787bfe\") " pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.734056 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.734022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4k8m\" (UniqueName: \"kubernetes.io/projected/01ed658a-d837-4ae6-a286-222f1cda2c28-kube-api-access-b4k8m\") pod \"multus-additional-cni-plugins-zdksc\" (UID: \"01ed658a-d837-4ae6-a286-222f1cda2c28\") " pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.734532 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.734509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnsd\" (UniqueName: \"kubernetes.io/projected/f6538151-c128-4020-98b4-0e9bb1959506-kube-api-access-gwnsd\") pod \"tuned-nprcv\" (UID: \"f6538151-c128-4020-98b4-0e9bb1959506\") " pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.734913 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.734886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6cq\" (UniqueName: \"kubernetes.io/projected/95bada68-c40f-4006-a489-e937d2475a6d-kube-api-access-db6cq\") pod \"iptables-alerter-vqr7r\" (UID: \"95bada68-c40f-4006-a489-e937d2475a6d\") " pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.736692 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.736675    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdj9c\" (UniqueName: \"kubernetes.io/projected/fd5179df-c0f5-4db1-a387-021f97d1cbd8-kube-api-access-vdj9c\") pod \"ovnkube-node-x9s78\" (UID: \"fd5179df-c0f5-4db1-a387-021f97d1cbd8\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.803174 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.803140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7"
Jul 28 12:35:09.813759 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.813681    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-js4c9"
Jul 28 12:35:09.816811 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.816766    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b84db21_8801_4bbd_b7a1_cdf05b764156.slice/crio-0d1e9f1f7e26b6f6f6473d6f0aef8573833bd90744fa3dc4453b69b55b5d3f74 WatchSource:0}: Error finding container 0d1e9f1f7e26b6f6f6473d6f0aef8573833bd90744fa3dc4453b69b55b5d3f74: Status 404 returned error can't find the container with id 0d1e9f1f7e26b6f6f6473d6f0aef8573833bd90744fa3dc4453b69b55b5d3f74
Jul 28 12:35:09.825633 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.825601    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9942989_507a_4a45_a294_217549de52ce.slice/crio-cff5ce047c9e020d38c31a4a2cc83a5999a00ad52905607ae6b8d54896365813 WatchSource:0}: Error finding container cff5ce047c9e020d38c31a4a2cc83a5999a00ad52905607ae6b8d54896365813: Status 404 returned error can't find the container with id cff5ce047c9e020d38c31a4a2cc83a5999a00ad52905607ae6b8d54896365813
Jul 28 12:35:09.825755 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.825702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kszkc"
Jul 28 12:35:09.835475 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.835450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zdksc"
Jul 28 12:35:09.837717 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.837671    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54e8191_d84d_430d_9bb1_1c5f76787bfe.slice/crio-b448d799814d134808ab7c9405761076c37899981b0899e639205343bd197d3a WatchSource:0}: Error finding container b448d799814d134808ab7c9405761076c37899981b0899e639205343bd197d3a: Status 404 returned error can't find the container with id b448d799814d134808ab7c9405761076c37899981b0899e639205343bd197d3a
Jul 28 12:35:09.839740 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.839711    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:35:09.845346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.845149    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ftf2m"
Jul 28 12:35:09.849442 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.849412    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ed658a_d837_4ae6_a286_222f1cda2c28.slice/crio-9f5bd066af14083ed52ed3e990408218ae69e11aa0c821649ec867a921712ef8 WatchSource:0}: Error finding container 9f5bd066af14083ed52ed3e990408218ae69e11aa0c821649ec867a921712ef8: Status 404 returned error can't find the container with id 9f5bd066af14083ed52ed3e990408218ae69e11aa0c821649ec867a921712ef8
Jul 28 12:35:09.854666 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.854645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:09.856241 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.856210    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a92a412_e8e8_4181_8d3e_d2388aea297d.slice/crio-3bd2b58f36143fbd8a1ebe423e530385ede76395eb0d2794c080d16244ca2e23 WatchSource:0}: Error finding container 3bd2b58f36143fbd8a1ebe423e530385ede76395eb0d2794c080d16244ca2e23: Status 404 returned error can't find the container with id 3bd2b58f36143fbd8a1ebe423e530385ede76395eb0d2794c080d16244ca2e23
Jul 28 12:35:09.863027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.863003    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nprcv"
Jul 28 12:35:09.866785 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.866754    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6342dafc_bb6e_4bda_9dcd_9afed70b8a6a.slice/crio-6fee17b0f8d5bbe30207872babad6ab05bd34ee7eefaec07e9ba6cb98c92875d WatchSource:0}: Error finding container 6fee17b0f8d5bbe30207872babad6ab05bd34ee7eefaec07e9ba6cb98c92875d: Status 404 returned error can't find the container with id 6fee17b0f8d5bbe30207872babad6ab05bd34ee7eefaec07e9ba6cb98c92875d
Jul 28 12:35:09.873799 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.873774    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6538151_c128_4020_98b4_0e9bb1959506.slice/crio-38ea3809cb2c92c4ca51c8abe144ac96d1e3f0ec7aa591f71e018f7d6afe36e8 WatchSource:0}: Error finding container 38ea3809cb2c92c4ca51c8abe144ac96d1e3f0ec7aa591f71e018f7d6afe36e8: Status 404 returned error can't find the container with id 38ea3809cb2c92c4ca51c8abe144ac96d1e3f0ec7aa591f71e018f7d6afe36e8
Jul 28 12:35:09.875028 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.875005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vqr7r"
Jul 28 12:35:09.882529 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:09.882501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:09.887752 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.887724    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bada68_c40f_4006_a489_e937d2475a6d.slice/crio-bef8ffb49e4a5d9fb5c545a0e821c1780a2468b61daf03d471d8239534456a65 WatchSource:0}: Error finding container bef8ffb49e4a5d9fb5c545a0e821c1780a2468b61daf03d471d8239534456a65: Status 404 returned error can't find the container with id bef8ffb49e4a5d9fb5c545a0e821c1780a2468b61daf03d471d8239534456a65
Jul 28 12:35:09.895151 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:35:09.895129    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5179df_c0f5_4db1_a387_021f97d1cbd8.slice/crio-f22fa8d66cfe63e5d8d61f0b861f6b3a593cc9ab3a7c61650a21780599defd48 WatchSource:0}: Error finding container f22fa8d66cfe63e5d8d61f0b861f6b3a593cc9ab3a7c61650a21780599defd48: Status 404 returned error can't find the container with id f22fa8d66cfe63e5d8d61f0b861f6b3a593cc9ab3a7c61650a21780599defd48
Jul 28 12:35:10.220141 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.220096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:10.220324 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.220208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:10.220324 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.220287    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:10.220324 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.220303    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:10.220518 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.220367    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:11.220346013 +0000 UTC m=+4.121490383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:10.220518 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.220390    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:11.220378023 +0000 UTC m=+4.121522393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:10.320960 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.320888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:10.321142 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.321089    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:10.321142 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.321115    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:10.321142 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.321135    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:10.321289 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.321197    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:11.321178962 +0000 UTC m=+4.222323332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:10.547137 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.546910    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:30:08 +0000 UTC" deadline="2027-12-25 02:00:22.426326034 +0000 UTC"
Jul 28 12:35:10.547137 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.546975    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12349h25m11.87935594s"
Jul 28 12:35:10.715565 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.715526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:10.715753 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:10.715680    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:10.734671 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.734628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nprcv" event={"ID":"f6538151-c128-4020-98b4-0e9bb1959506","Type":"ContainerStarted","Data":"38ea3809cb2c92c4ca51c8abe144ac96d1e3f0ec7aa591f71e018f7d6afe36e8"}
Jul 28 12:35:10.736355 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.736291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hmz9l" event={"ID":"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a","Type":"ContainerStarted","Data":"6fee17b0f8d5bbe30207872babad6ab05bd34ee7eefaec07e9ba6cb98c92875d"}
Jul 28 12:35:10.738490 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.738111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kszkc" event={"ID":"b54e8191-d84d-430d-9bb1-1c5f76787bfe","Type":"ContainerStarted","Data":"b448d799814d134808ab7c9405761076c37899981b0899e639205343bd197d3a"}
Jul 28 12:35:10.739811 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.739725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7" event={"ID":"7b84db21-8801-4bbd-b7a1-cdf05b764156","Type":"ContainerStarted","Data":"0d1e9f1f7e26b6f6f6473d6f0aef8573833bd90744fa3dc4453b69b55b5d3f74"}
Jul 28 12:35:10.741298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.741181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftf2m" event={"ID":"0a92a412-e8e8-4181-8d3e-d2388aea297d","Type":"ContainerStarted","Data":"3bd2b58f36143fbd8a1ebe423e530385ede76395eb0d2794c080d16244ca2e23"}
Jul 28 12:35:10.743271 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.742830    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerStarted","Data":"9f5bd066af14083ed52ed3e990408218ae69e11aa0c821649ec867a921712ef8"}
Jul 28 12:35:10.744415 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.744365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-js4c9" event={"ID":"f9942989-507a-4a45-a294-217549de52ce","Type":"ContainerStarted","Data":"cff5ce047c9e020d38c31a4a2cc83a5999a00ad52905607ae6b8d54896365813"}
Jul 28 12:35:10.745848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.745825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"f22fa8d66cfe63e5d8d61f0b861f6b3a593cc9ab3a7c61650a21780599defd48"}
Jul 28 12:35:10.747533 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:10.747488    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqr7r" event={"ID":"95bada68-c40f-4006-a489-e937d2475a6d","Type":"ContainerStarted","Data":"bef8ffb49e4a5d9fb5c545a0e821c1780a2468b61daf03d471d8239534456a65"}
Jul 28 12:35:11.229561 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:11.229526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:11.229717 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:11.229632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:11.229717 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.229654    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:11.229840 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.229743    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:13.229722046 +0000 UTC m=+6.130866436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:11.229840 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.229747    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:11.229840 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.229820    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:13.229803194 +0000 UTC m=+6.130947580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:11.331068 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:11.331031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:11.331248 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.331225    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:11.331318 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.331249    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:11.331318 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.331263    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:11.331415 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.331328    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:13.331305919 +0000 UTC m=+6.232450291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:11.715516 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:11.715128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:11.715516 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.715463    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:11.716119 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:11.715990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:11.726238 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:11.716849    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:11.754015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:11.753874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"d428843330d938518f1beb905f7443e6","Type":"ContainerStarted","Data":"4f0fac0ad5a79fd1f7702c1c83447097bd29d94908963a52befe2ad35177fe37"}
Jul 28 12:35:12.715314 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:12.715279    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:12.715548 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:12.715411    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:12.760323 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:12.759538    2576 generic.go:356] "Generic (PLEG): container finished" podID="d428843330d938518f1beb905f7443e6" containerID="4f0fac0ad5a79fd1f7702c1c83447097bd29d94908963a52befe2ad35177fe37" exitCode=0
Jul 28 12:35:12.760323 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:12.759597    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"d428843330d938518f1beb905f7443e6","Type":"ContainerDied","Data":"4f0fac0ad5a79fd1f7702c1c83447097bd29d94908963a52befe2ad35177fe37"}
Jul 28 12:35:13.249539 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:13.249493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:13.249753 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:13.249619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:13.249816 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.249748    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:13.249868 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.249822    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:17.249800902 +0000 UTC m=+10.150945274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:13.249868 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.249749    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:13.249975 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.249904    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:17.249884004 +0000 UTC m=+10.151028390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:13.350709 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:13.350665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:13.350897 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.350837    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:13.350897 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.350870    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:13.350897 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.350885    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:13.351064 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.350958    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:17.350936077 +0000 UTC m=+10.252080471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:13.715374 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:13.714791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:13.715374 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:13.714835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:13.715374 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.714951    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:13.715374 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:13.715044    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:14.714892 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:14.714856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:14.715078 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:14.715015    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:15.715000 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:15.714912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:15.715431 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:15.715066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:15.715588 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:15.715568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:15.715731 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:15.715694    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:16.715233 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:16.715201    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:16.715685 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:16.715339    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:17.280710 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:17.280672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:17.280901 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:17.280767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:17.280901 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.280844    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:17.280998 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.280913    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:25.280893066 +0000 UTC m=+18.182037448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:17.280998 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.280846    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:17.281106 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.281040    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:25.281014426 +0000 UTC m=+18.182158808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:17.381086 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:17.381050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:17.381283 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.381243    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:17.381283 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.381270    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:17.381283 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.381284    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:17.381440 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.381348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:25.381328213 +0000 UTC m=+18.282472596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:17.716015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:17.715859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:17.716015 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.715979    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:17.716015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:17.716020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:17.716633 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:17.716107    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:18.715537 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.715196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:18.715537 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:18.715327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:18.775487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.774742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"d428843330d938518f1beb905f7443e6","Type":"ContainerStarted","Data":"0b3e96e5e3ec10fbc427e46cd03405f8c62a31f7f90f6e6bedccb3ea65a7ad52"}
Jul 28 12:35:18.778432 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.778236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal" event={"ID":"1705df2f09f7b34e26c5e6a8a8046a32","Type":"ContainerStarted","Data":"a57b025589093cc9aaadcc7af271bdb0d75673bb9886dd213bbabaf243e2ff16"}
Jul 28 12:35:18.781635 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.781011    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ed658a-d837-4ae6-a286-222f1cda2c28" containerID="50b422ff38ee0e2f3ca69c996c7648a4d0d5e2c2eb583d2d1b390e78385f31e7" exitCode=0
Jul 28 12:35:18.781635 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.781081    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerDied","Data":"50b422ff38ee0e2f3ca69c996c7648a4d0d5e2c2eb583d2d1b390e78385f31e7"}
Jul 28 12:35:18.784484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.784055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-js4c9" event={"ID":"f9942989-507a-4a45-a294-217549de52ce","Type":"ContainerStarted","Data":"2acbc7130acbec7835d4c3a32d1c19d22b2dd05b8e0f32f4cb5e393686045840"}
Jul 28 12:35:18.786299 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.786190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nprcv" event={"ID":"f6538151-c128-4020-98b4-0e9bb1959506","Type":"ContainerStarted","Data":"0c51294fd09ee17a8eb86909aa14ea234c914008b2feb6ca6080f44f7727a9b4"}
Jul 28 12:35:18.788876 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.788358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hmz9l" event={"ID":"6342dafc-bb6e-4bda-9dcd-9afed70b8a6a","Type":"ContainerStarted","Data":"2aa83bd6a323a8f32e1e52e33bf95586c453fa0d4ec123b5782ccbc55d833f3f"}
Jul 28 12:35:18.791194 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.791120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kszkc" event={"ID":"b54e8191-d84d-430d-9bb1-1c5f76787bfe","Type":"ContainerStarted","Data":"d5e74268bd899638b7afc11e8ca5e28e54bfb35e961ae5b85db7ecf794c8e788"}
Jul 28 12:35:18.794324 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.794258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7" event={"ID":"7b84db21-8801-4bbd-b7a1-cdf05b764156","Type":"ContainerStarted","Data":"14a97df9527693e827b405f78a9b27023ae5914fd51e61f547fe1ec24bf3adbc"}
Jul 28 12:35:18.822407 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.822330    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" podStartSLOduration=10.82230425 podStartE2EDuration="10.82230425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.798094798 +0000 UTC m=+11.699239190" watchObservedRunningTime="2026-07-28 12:35:18.82230425 +0000 UTC m=+11.723448633"
Jul 28 12:35:18.857102 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.857041    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal" podStartSLOduration=10.857017895 podStartE2EDuration="10.857017895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.822815537 +0000 UTC m=+11.723959927" watchObservedRunningTime="2026-07-28 12:35:18.857017895 +0000 UTC m=+11.758162300"
Jul 28 12:35:18.894155 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.893115    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hmz9l" podStartSLOduration=2.921343864 podStartE2EDuration="10.893090855s" totalImagesPullingTime="7.971746991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.856973817 +0000 UTC m=+11.758118208" watchObservedRunningTime="2026-07-28 12:35:18.893090855 +0000 UTC m=+11.794235246"
Jul 28 12:35:18.920771 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.919950    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-js4c9" podStartSLOduration=2.88823315 podStartE2EDuration="10.9199263s" totalImagesPullingTime="8.03169315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.894017886 +0000 UTC m=+11.795162277" watchObservedRunningTime="2026-07-28 12:35:18.9199263 +0000 UTC m=+11.821070690"
Jul 28 12:35:18.945976 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.945926    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kszkc" podStartSLOduration=2.931811911 podStartE2EDuration="10.945906152s" totalImagesPullingTime="8.014094241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.920645733 +0000 UTC m=+11.821790124" watchObservedRunningTime="2026-07-28 12:35:18.945906152 +0000 UTC m=+11.847050543"
Jul 28 12:35:18.964600 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:18.964541    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nprcv" podStartSLOduration=2.963022854 podStartE2EDuration="10.964520686s" totalImagesPullingTime="8.001497832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:18.964134736 +0000 UTC m=+11.865279128" watchObservedRunningTime="2026-07-28 12:35:18.964520686 +0000 UTC m=+11.865665077"
Jul 28 12:35:19.715561 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:19.715105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:19.715561 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:19.715231    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:19.715561 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:19.715412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:19.715561 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:19.715553    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:19.803066 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:19.800459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vqr7r" event={"ID":"95bada68-c40f-4006-a489-e937d2475a6d","Type":"ContainerStarted","Data":"943f284683d19385b4f849cb044d772ab8e0077da9799f4b7ce4dea80dd80db8"}
Jul 28 12:35:19.820178 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:19.820129    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vqr7r" podStartSLOduration=3.855238684 podStartE2EDuration="11.820110617s" totalImagesPullingTime="7.964871933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:19.819638034 +0000 UTC m=+12.720782425" watchObservedRunningTime="2026-07-28 12:35:19.820110617 +0000 UTC m=+12.721255031"
Jul 28 12:35:19.862075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:19.862039    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:19.863023 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:19.862758    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:20.714658 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:20.714617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:20.714895 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:20.714770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:21.715130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:21.714816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:21.715130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:21.714830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:21.715130 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:21.714957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:21.715130 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:21.715034    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:21.804933 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:21.804903    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 12:35:21.826166 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:21.826134    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:21.826820 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:21.826790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hmz9l"
Jul 28 12:35:22.715044 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:22.715006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:22.715269 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:22.715144    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:23.715789 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:23.715276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:23.715789 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:23.715434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:23.715789 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:23.715744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:23.716479 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:23.715852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:24.430057 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:24.430038    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:35:24.561783 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:24.561666    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:35:24.430053797Z","UUID":"4a118154-a890-4c15-95b0-06f297d86c1c","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:35:24.563418 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:24.563396    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:35:24.563561 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:24.563444    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:35:24.715049 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:24.714968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:24.715208 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:24.715110    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:24.814293 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:24.814249    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7" event={"ID":"7b84db21-8801-4bbd-b7a1-cdf05b764156","Type":"ContainerStarted","Data":"6451a2f3200fc5f2874beab6d3a69336843304aa5c8fd2d20eeffb085463f21a"}
Jul 28 12:35:25.345096 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:25.344839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:25.345388 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:25.345120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:25.345388 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.345016    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:25.345388 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.345242    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:25.345388 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.345254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:41.345222651 +0000 UTC m=+34.246367022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:25.345388 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.345293    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:41.345277563 +0000 UTC m=+34.246421935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:25.445614 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:25.445473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:25.445614 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.445620    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:25.445812 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.445638    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:25.445812 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.445650    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:25.445812 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.445708    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:35:41.445690008 +0000 UTC m=+34.346834376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:25.715326 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:25.715242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:25.715921 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.715389    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:25.715921 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:25.715401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:25.715921 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:25.715523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:26.714708 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:26.714676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:26.715116 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:26.714795    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:26.796914 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:26.796882    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-js4c9_f9942989-507a-4a45-a294-217549de52ce/dns-node-resolver/0.log"
Jul 28 12:35:27.716092 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:27.716063    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:27.716550 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:27.716165    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:27.716550 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:27.716247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:27.716550 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:27.716352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:27.776014 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:27.775988    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kszkc_b54e8191-d84d-430d-9bb1-1c5f76787bfe/node-ca/0.log"
Jul 28 12:35:28.714727 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:28.714699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:28.714883 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:28.714798    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:29.714875 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:29.714841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:29.715354 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:29.714851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:29.715354 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:29.714957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:29.715354 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:29.715027    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:29.826150 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:29.826056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ftf2m" event={"ID":"0a92a412-e8e8-4181-8d3e-d2388aea297d","Type":"ContainerStarted","Data":"de576e170f3d918dd5a664f26360895531b9d355a345e9df31c84ead7c88a2d9"}
Jul 28 12:35:29.827701 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:29.827661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"0b5faf027893191c89cdcc19622c676f587c73939dc6f92622e54b0b7d588d3b"}
Jul 28 12:35:29.851947 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:29.851890    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ftf2m" podStartSLOduration=2.880409111 podStartE2EDuration="21.85187112s" totalImagesPullingTime="18.971462009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:29.850905026 +0000 UTC m=+22.752049417" watchObservedRunningTime="2026-07-28 12:35:29.85187112 +0000 UTC m=+22.753015511"
Jul 28 12:35:30.715103 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:30.715070    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:30.715586 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:30.715189    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:31.715257 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.715017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:31.715625 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.715086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:31.715625 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:31.715361    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:31.715625 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:31.715471    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:31.833826 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.833789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7" event={"ID":"7b84db21-8801-4bbd-b7a1-cdf05b764156","Type":"ContainerStarted","Data":"494d71243aea93600d283ab0e90e5a3d3c4d29b080e19105bee147d5003ffe11"}
Jul 28 12:35:31.835535 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.835504    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ed658a-d837-4ae6-a286-222f1cda2c28" containerID="5bd5cd2f03d6df73d7d02dd63a7a8e4a261e52a6e3fbf000e609735c36fe714a" exitCode=0
Jul 28 12:35:31.835649 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.835580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerDied","Data":"5bd5cd2f03d6df73d7d02dd63a7a8e4a261e52a6e3fbf000e609735c36fe714a"}
Jul 28 12:35:31.839016 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.838996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"626af162a9e051e915362c383c9699bca55a056fbf2aa39a374fc71e4ccd4a1e"}
Jul 28 12:35:31.839103 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.839051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"09c89d66f6cac5533e98eab3daba7963a1b9cd56cedf4c4c341dec737eaa62b3"}
Jul 28 12:35:31.839103 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.839082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"a876fc66965934b103ee6303562a8406467f76fd25784c036e52d1261dce1693"}
Jul 28 12:35:31.839182 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.839110    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"53c3ef249be03e91a8b39da857067f0cfe78b3088ecdc0f6a89738f39d57872e"}
Jul 28 12:35:31.839182 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.839138    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"696f6394de9e37c7a6541a409d9aba658145b2ff667a286a11328e59d216903e"}
Jul 28 12:35:31.853270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:31.853217    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4t7k7" podStartSLOduration=2.746128668 podStartE2EDuration="23.853205613s" totalImagesPullingTime="21.107076945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:31.851908636 +0000 UTC m=+24.753053025" watchObservedRunningTime="2026-07-28 12:35:31.853205613 +0000 UTC m=+24.754349999"
Jul 28 12:35:32.714911 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:32.714881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:32.715059 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:32.714975    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:33.715351 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:33.715290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:33.715782 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:33.715406    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:33.715782 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:33.715443    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:33.715782 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:33.715492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:33.845315 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:33.845292    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ed658a-d837-4ae6-a286-222f1cda2c28" containerID="b479532d89fd6121c47a9c06f353309ee051d1fc36e2b754264e6d1d2c6296c2" exitCode=0
Jul 28 12:35:33.845438 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:33.845341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerDied","Data":"b479532d89fd6121c47a9c06f353309ee051d1fc36e2b754264e6d1d2c6296c2"}
Jul 28 12:35:34.714706 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:34.714633    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:34.714830 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:34.714741    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:34.849838 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:34.849804    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ed658a-d837-4ae6-a286-222f1cda2c28" containerID="f68b787ec206957d8419cb405e717849d246278834bfb9269f5108b50c161083" exitCode=0
Jul 28 12:35:34.850304 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:34.849885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerDied","Data":"f68b787ec206957d8419cb405e717849d246278834bfb9269f5108b50c161083"}
Jul 28 12:35:34.853231 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:34.853212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"fd38e6ff8841c8557fbd89fc8a6be55a819b1b7f97aba016c057bac1acdd609f"}
Jul 28 12:35:35.715598 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:35.715501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:35.715598 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:35.715516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:35.715785 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:35.715618    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:35.715785 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:35.715727    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:36.715496 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:36.715281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:36.716088 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:36.715595    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:36.862431 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:36.862390    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" event={"ID":"fd5179df-c0f5-4db1-a387-021f97d1cbd8","Type":"ContainerStarted","Data":"ee2d4fa4d1303e7c7b4633b7e18f2acead1efa067927e5ec81217c0fbe580ade"}
Jul 28 12:35:36.862708 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:36.862667    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:36.862846 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:36.862756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:36.881329 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:36.881305    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:36.892044 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:36.892003    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78" podStartSLOduration=9.619026195 podStartE2EDuration="28.891988978s" totalImagesPullingTime="19.272962783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:36.891593496 +0000 UTC m=+29.792737908" watchObservedRunningTime="2026-07-28 12:35:36.891988978 +0000 UTC m=+29.793133372"
Jul 28 12:35:37.718078 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:37.718044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:37.718499 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:37.718172    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:37.718499 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:37.718223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:37.718499 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:37.718330    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:37.865763 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:37.865735    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:37.881270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:37.881245    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:35:38.715435 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:38.715405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:38.715590 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:38.715510    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:39.717781 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:39.717747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:39.718141 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:39.717861    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:39.718141 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:39.717890    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:39.718141 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:39.717948    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:39.871912 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:39.871877    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ed658a-d837-4ae6-a286-222f1cda2c28" containerID="3ed1e39e87af891773fbcccf45b5c18da463e5f4b2312dfad34bc88f0a5025c0" exitCode=0
Jul 28 12:35:39.872012 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:39.871967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerDied","Data":"3ed1e39e87af891773fbcccf45b5c18da463e5f4b2312dfad34bc88f0a5025c0"}
Jul 28 12:35:40.714510 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:40.714482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:40.714662 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:40.714582    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:40.878287 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:40.878251    2576 generic.go:356] "Generic (PLEG): container finished" podID="01ed658a-d837-4ae6-a286-222f1cda2c28" containerID="9a4ea0a4dc45fdf8c03950fff1828e81bfdb934eca959dc5b6d2992d40f52feb" exitCode=0
Jul 28 12:35:40.878664 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:40.878320    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerDied","Data":"9a4ea0a4dc45fdf8c03950fff1828e81bfdb934eca959dc5b6d2992d40f52feb"}
Jul 28 12:35:41.365426 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.365397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:41.365606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.365436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:41.365606 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.365530    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:41.365606 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.365536    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:41.365606 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.365577    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret podName:5352b7da-d4d1-4cca-9d6e-d522d0495f42 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:13.365563656 +0000 UTC m=+66.266708025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret") pod "global-pull-secret-syncer-rg6kq" (UID: "5352b7da-d4d1-4cca-9d6e-d522d0495f42") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:35:41.365606 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.365593    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs podName:c80ac7ce-b3d4-481e-8c0f-6486a5a44827 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:13.36558538 +0000 UTC m=+66.266729748 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs") pod "network-metrics-daemon-hcvxx" (UID: "c80ac7ce-b3d4-481e-8c0f-6486a5a44827") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:35:41.466058 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.466034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:41.466189 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.466173    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:35:41.466233 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.466192    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:35:41.466233 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.466202    2576 projected.go:196] Error preparing data for projected volume kube-api-access-n5rq9 for pod openshift-network-diagnostics/network-check-target-h9c6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:41.466298 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.466245    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9 podName:ce31f5aa-2e18-4923-8c01-a630b25d1e52 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:13.466231107 +0000 UTC m=+66.367375475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n5rq9" (UniqueName: "kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9") pod "network-check-target-h9c6h" (UID: "ce31f5aa-2e18-4923-8c01-a630b25d1e52") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:35:41.715030 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.714972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:41.715030 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.715001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:41.715287 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.715063    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:41.715287 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:41.715194    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:41.884058 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.884034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zdksc" event={"ID":"01ed658a-d837-4ae6-a286-222f1cda2c28","Type":"ContainerStarted","Data":"315f99d140f8425228110efca39504f933db03d61c4ff99bd3cb58b1e38ed904"}
Jul 28 12:35:41.909060 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:41.909011    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zdksc" podStartSLOduration=6.970218839 podStartE2EDuration="33.908998531s" totalImagesPullingTime="26.938779692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:35:41.907641153 +0000 UTC m=+34.808785544" watchObservedRunningTime="2026-07-28 12:35:41.908998531 +0000 UTC m=+34.810142921"
Jul 28 12:35:42.715408 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:42.715364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:42.715569 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:42.715468    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:43.715200 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:43.715166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:43.715640 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:43.715290    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:43.715640 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:43.715360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:43.715640 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:43.715471    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:44.715015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:44.714986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:44.715181 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:44.715077    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:45.715224 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:45.715191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:45.719212 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:45.719173    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:45.719212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:45.719181    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:45.719439 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:45.719414    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:46.714731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:46.714702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:46.714894 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:46.714801    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:47.716140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:47.716112    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:47.716493 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:47.716187    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:47.716493 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:47.716199    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:47.716493 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:47.716291    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:48.714870 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:48.714841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:48.715040 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:48.714950    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:49.715217 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:49.715180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:49.715217 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:49.715206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:49.715758 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:49.715296    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:49.715758 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:49.715445    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:50.715342 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:50.715309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:50.715771 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:50.715425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:51.714962 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:51.714932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:51.715138 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:51.715040    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:51.715212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:51.715136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:51.715298 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:51.715274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:52.714712 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:52.714685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:52.715081 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:52.714773    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:53.714571 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:53.714547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:53.714768 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:53.714664    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:53.714768 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:53.714704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:53.715108 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:53.714791    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:54.714501 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:54.714466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:54.714656 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:54.714567    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:55.714737 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:55.714706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:55.715107 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:55.714706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:55.715107 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:55.714822    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:55.715107 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:55.714868    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:56.715491 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:56.715463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:56.715848 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:56.715555    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:57.356027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:57.355872    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rg6kq"]
Jul 28 12:35:57.356147 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:57.356098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:57.356211 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:57.356194    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:57.359628 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:57.359595    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h9c6h"]
Jul 28 12:35:57.359751 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:57.359706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:57.359817 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:57.359800    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:57.377326 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:57.377305    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hcvxx"]
Jul 28 12:35:57.377440 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:57.377415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:57.377518 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:57.377496    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:35:58.714820 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:58.714791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:35:58.715329 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:58.714887    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:35:59.714939 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:59.714905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:35:59.715433 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:59.715026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9c6h" podUID="ce31f5aa-2e18-4923-8c01-a630b25d1e52"
Jul 28 12:35:59.715433 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:35:59.715086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:35:59.715433 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:35:59.715183    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hcvxx" podUID="c80ac7ce-b3d4-481e-8c0f-6486a5a44827"
Jul 28 12:36:00.714808 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:00.714782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:36:00.714981 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:00.714872    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rg6kq" podUID="5352b7da-d4d1-4cca-9d6e-d522d0495f42"
Jul 28 12:36:01.423397 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.423353    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeReady"
Jul 28 12:36:01.423512 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.423471    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:36:01.499414 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.499347    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v6s4d"]
Jul 28 12:36:01.515614 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.515595    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kf94z"]
Jul 28 12:36:01.515782 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.515749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.518828 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.518806    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:36:01.519059 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.519036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s2w2r\""
Jul 28 12:36:01.519150 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.519091    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:36:01.519150 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.519114    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:36:01.539479 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.539459    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v6s4d"]
Jul 28 12:36:01.539579 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.539492    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kf94z"]
Jul 28 12:36:01.539638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.539601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.542620 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.542602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:36:01.542815 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.542801    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:36:01.542939 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.542924    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:36:01.543036 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.543022    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-64vgn\""
Jul 28 12:36:01.543911 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.543876    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:36:01.591682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.591655    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-s9nqg"]
Jul 28 12:36:01.611735 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87172efd-f000-4b9c-8019-f1e930b2a804-crio-socket\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.611848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8b1764d-7728-4e13-99f2-1d17c4d4d896-cert\") pod \"ingress-canary-v6s4d\" (UID: \"d8b1764d-7728-4e13-99f2-1d17c4d4d896\") " pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.611848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611769    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87172efd-f000-4b9c-8019-f1e930b2a804-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.611848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glfcn\" (UniqueName: \"kubernetes.io/projected/87172efd-f000-4b9c-8019-f1e930b2a804-kube-api-access-glfcn\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.611848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqb4\" (UniqueName: \"kubernetes.io/projected/d8b1764d-7728-4e13-99f2-1d17c4d4d896-kube-api-access-8lqb4\") pod \"ingress-canary-v6s4d\" (UID: \"d8b1764d-7728-4e13-99f2-1d17c4d4d896\") " pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.611998 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87172efd-f000-4b9c-8019-f1e930b2a804-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.611998 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.611900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87172efd-f000-4b9c-8019-f1e930b2a804-data-volume\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.615849 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.615834    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s9nqg"]
Jul 28 12:36:01.615914 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.615904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.618932 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.618909    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4g9qb\""
Jul 28 12:36:01.619022 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.618997    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:36:01.619085 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.619018    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:36:01.712677 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87172efd-f000-4b9c-8019-f1e930b2a804-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.712765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdca270-27e6-432b-b86e-2d233993745c-config-volume\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.712765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glfcn\" (UniqueName: \"kubernetes.io/projected/87172efd-f000-4b9c-8019-f1e930b2a804-kube-api-access-glfcn\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.712765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqb4\" (UniqueName: \"kubernetes.io/projected/d8b1764d-7728-4e13-99f2-1d17c4d4d896-kube-api-access-8lqb4\") pod \"ingress-canary-v6s4d\" (UID: \"d8b1764d-7728-4e13-99f2-1d17c4d4d896\") " pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.712765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87172efd-f000-4b9c-8019-f1e930b2a804-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.712940 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6747\" (UniqueName: \"kubernetes.io/projected/9cdca270-27e6-432b-b86e-2d233993745c-kube-api-access-h6747\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.712940 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87172efd-f000-4b9c-8019-f1e930b2a804-data-volume\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.712940 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87172efd-f000-4b9c-8019-f1e930b2a804-crio-socket\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.712940 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8b1764d-7728-4e13-99f2-1d17c4d4d896-cert\") pod \"ingress-canary-v6s4d\" (UID: \"d8b1764d-7728-4e13-99f2-1d17c4d4d896\") " pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.713159 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca270-27e6-432b-b86e-2d233993745c-tmp-dir\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.713159 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.712997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cdca270-27e6-432b-b86e-2d233993745c-metrics-tls\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.713159 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.713073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87172efd-f000-4b9c-8019-f1e930b2a804-crio-socket\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.713159 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.713126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87172efd-f000-4b9c-8019-f1e930b2a804-data-volume\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.713418 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.713398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87172efd-f000-4b9c-8019-f1e930b2a804-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.715569 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.715546    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:36:01.716013 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.715720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:36:01.716649 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.716621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87172efd-f000-4b9c-8019-f1e930b2a804-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.716724 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.716665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8b1764d-7728-4e13-99f2-1d17c4d4d896-cert\") pod \"ingress-canary-v6s4d\" (UID: \"d8b1764d-7728-4e13-99f2-1d17c4d4d896\") " pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.720749 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.720732    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:36:01.720847 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.720828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:36:01.720919 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.720822    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:36:01.720970 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.720752    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf2p5\""
Jul 28 12:36:01.721076 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.721053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc56m\""
Jul 28 12:36:01.724076 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.724053    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glfcn\" (UniqueName: \"kubernetes.io/projected/87172efd-f000-4b9c-8019-f1e930b2a804-kube-api-access-glfcn\") pod \"insights-runtime-extractor-kf94z\" (UID: \"87172efd-f000-4b9c-8019-f1e930b2a804\") " pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.724171 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.724156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqb4\" (UniqueName: \"kubernetes.io/projected/d8b1764d-7728-4e13-99f2-1d17c4d4d896-kube-api-access-8lqb4\") pod \"ingress-canary-v6s4d\" (UID: \"d8b1764d-7728-4e13-99f2-1d17c4d4d896\") " pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.813502 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.813459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca270-27e6-432b-b86e-2d233993745c-tmp-dir\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.813502 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.813485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cdca270-27e6-432b-b86e-2d233993745c-metrics-tls\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.813639 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.813509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdca270-27e6-432b-b86e-2d233993745c-config-volume\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.813639 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.813545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6747\" (UniqueName: \"kubernetes.io/projected/9cdca270-27e6-432b-b86e-2d233993745c-kube-api-access-h6747\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.813786 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.813768    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cdca270-27e6-432b-b86e-2d233993745c-tmp-dir\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.814031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.814010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdca270-27e6-432b-b86e-2d233993745c-config-volume\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.815452 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.815433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cdca270-27e6-432b-b86e-2d233993745c-metrics-tls\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.821798 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.821781    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6747\" (UniqueName: \"kubernetes.io/projected/9cdca270-27e6-432b-b86e-2d233993745c-kube-api-access-h6747\") pod \"dns-default-s9nqg\" (UID: \"9cdca270-27e6-432b-b86e-2d233993745c\") " pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:01.826712 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.826690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v6s4d"
Jul 28 12:36:01.850243 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.850226    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kf94z"
Jul 28 12:36:01.923254 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:01.923053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:02.007388 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.007336    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v6s4d"]
Jul 28 12:36:02.011143 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.011118    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kf94z"]
Jul 28 12:36:02.014568 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:02.014539    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87172efd_f000_4b9c_8019_f1e930b2a804.slice/crio-65f35b24f1c1acdb2f3b54c29a8cfc07d8e3145f764e355fea4a0f0832482026 WatchSource:0}: Error finding container 65f35b24f1c1acdb2f3b54c29a8cfc07d8e3145f764e355fea4a0f0832482026: Status 404 returned error can't find the container with id 65f35b24f1c1acdb2f3b54c29a8cfc07d8e3145f764e355fea4a0f0832482026
Jul 28 12:36:02.060058 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.060030    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-s9nqg"]
Jul 28 12:36:02.063490 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:02.063468    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdca270_27e6_432b_b86e_2d233993745c.slice/crio-ec6e3d8312471e0d4cc7f73a1daf0e6a7ebe04c00b6d566637f7443ccdd98fc0 WatchSource:0}: Error finding container ec6e3d8312471e0d4cc7f73a1daf0e6a7ebe04c00b6d566637f7443ccdd98fc0: Status 404 returned error can't find the container with id ec6e3d8312471e0d4cc7f73a1daf0e6a7ebe04c00b6d566637f7443ccdd98fc0
Jul 28 12:36:02.715489 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.715458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:36:02.718687 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.718662    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:36:02.928359 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.928323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s9nqg" event={"ID":"9cdca270-27e6-432b-b86e-2d233993745c","Type":"ContainerStarted","Data":"ec6e3d8312471e0d4cc7f73a1daf0e6a7ebe04c00b6d566637f7443ccdd98fc0"}
Jul 28 12:36:02.929688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.929657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v6s4d" event={"ID":"d8b1764d-7728-4e13-99f2-1d17c4d4d896","Type":"ContainerStarted","Data":"614c785286ea1cde5fa10762815c274b4be903b4ae4d4892202b23fcfe9544b4"}
Jul 28 12:36:02.930840 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:02.930820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf94z" event={"ID":"87172efd-f000-4b9c-8019-f1e930b2a804","Type":"ContainerStarted","Data":"65f35b24f1c1acdb2f3b54c29a8cfc07d8e3145f764e355fea4a0f0832482026"}
Jul 28 12:36:06.223994 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.223963    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85f98d57d6-zb7xk"]
Jul 28 12:36:06.241347 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.241319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f98d57d6-zb7xk"]
Jul 28 12:36:06.241512 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.241449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.244385 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.244347    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-924zr\""
Jul 28 12:36:06.244832 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.244661    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:36:06.244832 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.244764    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:36:06.244832 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.244774    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:36:06.245679 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.245647    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:36:06.245821 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.245680    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:36:06.245974 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.245953    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:36:06.246083 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.245954    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:36:06.251975 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.251949    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:36:06.347417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-serving-cert\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.347549 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-oauth-serving-cert\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.347549 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-service-ca\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.347549 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-oauth-config\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.347692 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-trusted-ca-bundle\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.347692 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-config\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.347692 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.347662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwq8f\" (UniqueName: \"kubernetes.io/projected/cd61ac96-302b-454d-a9c1-d53aa35309fa-kube-api-access-dwq8f\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.448727 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.448695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-oauth-config\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.448869 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.448760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-trusted-ca-bundle\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.448869 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.448788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-config\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.448869 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.448814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwq8f\" (UniqueName: \"kubernetes.io/projected/cd61ac96-302b-454d-a9c1-d53aa35309fa-kube-api-access-dwq8f\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449049 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-serving-cert\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449110 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-oauth-serving-cert\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449176 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-service-ca\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449593 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449564    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-config\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449693 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-oauth-serving-cert\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-trusted-ca-bundle\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.449789 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.449776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-service-ca\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.452723 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.452701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-oauth-config\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.452723 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.452712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-serving-cert\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.463835 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.463815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwq8f\" (UniqueName: \"kubernetes.io/projected/cd61ac96-302b-454d-a9c1-d53aa35309fa-kube-api-access-dwq8f\") pod \"console-85f98d57d6-zb7xk\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") " pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:06.552790 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:06.552716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:07.659085 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.659059    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"]
Jul 28 12:36:07.662297 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.662280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.666992 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.666974    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mdn25\""
Jul 28 12:36:07.666992 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.666983    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:36:07.668355 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.668329    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:36:07.668837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.668572    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:36:07.668837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.668753    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 12:36:07.668983 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.668965    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:36:07.678000 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.677943    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"]
Jul 28 12:36:07.693384 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.693352    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l6shh"]
Jul 28 12:36:07.696517 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.696497    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.699334 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.699312    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:36:07.699911 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.699891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g65q6\""
Jul 28 12:36:07.700011 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.699951    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:36:07.701255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.701237    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:36:07.739382 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.739336    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"]
Jul 28 12:36:07.742357 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.742334    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.745468 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.745444    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 12:36:07.745996 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.745841    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 12:36:07.745996 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.745907    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 12:36:07.745996 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.745941    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5cpfb\""
Jul 28 12:36:07.758428 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-textfile\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758527 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758527 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-wtmp\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758527 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbvr\" (UniqueName: \"kubernetes.io/projected/e679bdb4-1b51-4f64-b3b3-2567f68d030d-kube-api-access-rvbvr\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.758690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.758690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.758690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b7d56b-f47c-48a5-b2b9-b26451aee178-metrics-client-ca\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e679bdb4-1b51-4f64-b3b3-2567f68d030d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.758981 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-sys\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758981 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-root\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758981 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-accelerators-collector-config\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.758981 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.758848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpbk\" (UniqueName: \"kubernetes.io/projected/51b7d56b-f47c-48a5-b2b9-b26451aee178-kube-api-access-svpbk\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.767064 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.767043    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"]
Jul 28 12:36:07.860656 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-textfile\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.860656 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jth2d\" (UniqueName: \"kubernetes.io/projected/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-api-access-jth2d\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-wtmp\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbvr\" (UniqueName: \"kubernetes.io/projected/e679bdb4-1b51-4f64-b3b3-2567f68d030d-kube-api-access-rvbvr\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.860915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.860957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b7d56b-f47c-48a5-b2b9-b26451aee178-metrics-client-ca\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e679bdb4-1b51-4f64-b3b3-2567f68d030d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-sys\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-root\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-accelerators-collector-config\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpbk\" (UniqueName: \"kubernetes.io/projected/51b7d56b-f47c-48a5-b2b9-b26451aee178-kube-api-access-svpbk\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-textfile\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:07.861513    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:07.861566    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls podName:51b7d56b-f47c-48a5-b2b9-b26451aee178 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.361546572 +0000 UTC m=+61.262690956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls") pod "node-exporter-l6shh" (UID: "51b7d56b-f47c-48a5-b2b9-b26451aee178") : secret "node-exporter-tls" not found
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861671    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-wtmp\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-sys\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:07.861759    2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 12:36:07.861794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.861793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b7d56b-f47c-48a5-b2b9-b26451aee178-root\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.862139 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:07.861831    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-tls podName:e679bdb4-1b51-4f64-b3b3-2567f68d030d nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.361814584 +0000 UTC m=+61.262958955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-6r5gw" (UID: "e679bdb4-1b51-4f64-b3b3-2567f68d030d") : secret "openshift-state-metrics-tls" not found
Jul 28 12:36:07.864270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.862299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-accelerators-collector-config\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.864270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.862331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e679bdb4-1b51-4f64-b3b3-2567f68d030d-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.864270 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.862472    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b7d56b-f47c-48a5-b2b9-b26451aee178-metrics-client-ca\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.869739 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.865828    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.870867 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.870822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.886463 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.886415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbvr\" (UniqueName: \"kubernetes.io/projected/e679bdb4-1b51-4f64-b3b3-2567f68d030d-kube-api-access-rvbvr\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:07.886569 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.886475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpbk\" (UniqueName: \"kubernetes.io/projected/51b7d56b-f47c-48a5-b2b9-b26451aee178-kube-api-access-svpbk\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:07.962546 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.962519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jth2d\" (UniqueName: \"kubernetes.io/projected/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-api-access-jth2d\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.962654 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.962579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.962654 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.962631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.962769 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.962693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.962769 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.962728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.962870 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.962769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.962924 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:07.962910    2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 12:36:07.962975 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:07.962970    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-tls podName:5c2f3458-2fa4-49b5-9c3e-691b1e6276ec nodeName:}" failed. No retries permitted until 2026-07-28 12:36:08.462948961 +0000 UTC m=+61.364093349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-rpv9f" (UID: "5c2f3458-2fa4-49b5-9c3e-691b1e6276ec") : secret "kube-state-metrics-tls" not found
Jul 28 12:36:07.964477 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.963629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.964477 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.963710    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.964477 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.964065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.966648 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.966624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:07.985904 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:07.985881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jth2d\" (UniqueName: \"kubernetes.io/projected/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-api-access-jth2d\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:08.019233 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.019018    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f98d57d6-zb7xk"]
Jul 28 12:36:08.024102 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:08.024079    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd61ac96_302b_454d_a9c1_d53aa35309fa.slice/crio-d8f55775c6db12d800d94b3a7492e4f7daab120b05d653d46621e610db8d5894 WatchSource:0}: Error finding container d8f55775c6db12d800d94b3a7492e4f7daab120b05d653d46621e610db8d5894: Status 404 returned error can't find the container with id d8f55775c6db12d800d94b3a7492e4f7daab120b05d653d46621e610db8d5894
Jul 28 12:36:08.366479 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.366445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:08.366636 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.366497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:08.366636 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:08.366595    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:36:08.366700 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:08.366657    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls podName:51b7d56b-f47c-48a5-b2b9-b26451aee178 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:09.366641522 +0000 UTC m=+62.267785891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls") pod "node-exporter-l6shh" (UID: "51b7d56b-f47c-48a5-b2b9-b26451aee178") : secret "node-exporter-tls" not found
Jul 28 12:36:08.368893 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.368867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e679bdb4-1b51-4f64-b3b3-2567f68d030d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6r5gw\" (UID: \"e679bdb4-1b51-4f64-b3b3-2567f68d030d\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:08.466779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.466755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:08.468865 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.468849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c2f3458-2fa4-49b5-9c3e-691b1e6276ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-rpv9f\" (UID: \"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:08.570990 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.570966    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"
Jul 28 12:36:08.654549 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.654514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"
Jul 28 12:36:08.754069 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.753983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw"]
Jul 28 12:36:08.756940 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:08.756910    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode679bdb4_1b51_4f64_b3b3_2567f68d030d.slice/crio-57b0341571d0aa46a1a853661938398963017c4119e2cf0019a477e389f7e3b6 WatchSource:0}: Error finding container 57b0341571d0aa46a1a853661938398963017c4119e2cf0019a477e389f7e3b6: Status 404 returned error can't find the container with id 57b0341571d0aa46a1a853661938398963017c4119e2cf0019a477e389f7e3b6
Jul 28 12:36:08.831526 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.831500    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f"]
Jul 28 12:36:08.836657 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:08.836627    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2f3458_2fa4_49b5_9c3e_691b1e6276ec.slice/crio-17701663d43b6b8b1e439f6d4ae3b7521bd6c9f7183311a13ab0cf02921c6035 WatchSource:0}: Error finding container 17701663d43b6b8b1e439f6d4ae3b7521bd6c9f7183311a13ab0cf02921c6035: Status 404 returned error can't find the container with id 17701663d43b6b8b1e439f6d4ae3b7521bd6c9f7183311a13ab0cf02921c6035
Jul 28 12:36:08.897498 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.897476    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:08.912527 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.912507    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.918389 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.917939    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:36:08.918389 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.918179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:36:08.918389 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.918361    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:36:08.918580 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.918544    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:36:08.918580 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.918564    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:36:08.918762 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.918745    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:36:08.918894 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.918804    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:36:08.919017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.919001    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:36:08.919083 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.919026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:36:08.919272 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.919253    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dbs7f\""
Jul 28 12:36:08.941126 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.941048    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:08.954735 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.954712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s9nqg" event={"ID":"9cdca270-27e6-432b-b86e-2d233993745c","Type":"ContainerStarted","Data":"71991249c658af9314ee3dd21d9dd9a7f59effd3cce811895dfc4b4e31b55c10"}
Jul 28 12:36:08.954891 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.954756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-s9nqg" event={"ID":"9cdca270-27e6-432b-b86e-2d233993745c","Type":"ContainerStarted","Data":"b5c19a40a94cbd48ee13bca35baeb937bffb01a1ba26a47f6f1de0ddcebab95d"}
Jul 28 12:36:08.954891 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.954786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:08.957907 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.957556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v6s4d" event={"ID":"d8b1764d-7728-4e13-99f2-1d17c4d4d896","Type":"ContainerStarted","Data":"b22e028fc0ec1136317512f1257637cf8ba7be18891c06803f5112ee1ca67815"}
Jul 28 12:36:08.959505 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.959287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf94z" event={"ID":"87172efd-f000-4b9c-8019-f1e930b2a804","Type":"ContainerStarted","Data":"f2cc7f8167d8e1c7940ba2d5267252fd4804bd0b280087a67b11defb27e15176"}
Jul 28 12:36:08.961401 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.961344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw" event={"ID":"e679bdb4-1b51-4f64-b3b3-2567f68d030d","Type":"ContainerStarted","Data":"a7b51ac293544f0fd8cb0352647f619e842843ee19e1f6c0770f05459e4fec06"}
Jul 28 12:36:08.961506 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.961416    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw" event={"ID":"e679bdb4-1b51-4f64-b3b3-2567f68d030d","Type":"ContainerStarted","Data":"57b0341571d0aa46a1a853661938398963017c4119e2cf0019a477e389f7e3b6"}
Jul 28 12:36:08.962910 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.962809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f" event={"ID":"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec","Type":"ContainerStarted","Data":"17701663d43b6b8b1e439f6d4ae3b7521bd6c9f7183311a13ab0cf02921c6035"}
Jul 28 12:36:08.964198 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.964176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f98d57d6-zb7xk" event={"ID":"cd61ac96-302b-454d-a9c1-d53aa35309fa","Type":"ContainerStarted","Data":"d8f55775c6db12d800d94b3a7492e4f7daab120b05d653d46621e610db8d5894"}
Jul 28 12:36:08.971693 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.971800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-web-config\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.971800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.971800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-config-volume\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.971800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.971950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-config-out\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.972037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.972082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.972100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.972122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.972155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcw2\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-kube-api-access-shcw2\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:08.972251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:08.972173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.021452 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.020190    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-s9nqg" podStartSLOduration=4.880317985 podStartE2EDuration="8.020170198s" totalImagesPullingTime="3.139852213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:09.019713724 +0000 UTC m=+61.920858117" watchObservedRunningTime="2026-07-28 12:36:09.020170198 +0000 UTC m=+61.921314589"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.073751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-config-out\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.073795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.073871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.073902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.073930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:09.073983    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle podName:5784be4c-eebd-4ee5-83dd-fcf383439637 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:09.573959069 +0000 UTC m=+62.475103442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637") : configmap references non-existent config key: ca-bundle.crt
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shcw2\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-kube-api-access-shcw2\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074986 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074986 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-web-config\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074986 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074986 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-config-volume\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.074986 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.074976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.075250 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.075029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.075250 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:09.075140    2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 28 12:36:09.075250 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:09.075190    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls podName:5784be4c-eebd-4ee5-83dd-fcf383439637 nodeName:}" failed. No retries permitted until 2026-07-28 12:36:09.575173808 +0000 UTC m=+62.476318176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637") : secret "alertmanager-main-tls" not found
Jul 28 12:36:09.076114 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.075801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.076114 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.075863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.077731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.077706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.079584 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.079288    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.079816 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.079771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.079919 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.079849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-config-volume\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.080413 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.080301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.080413 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.080422    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-config-out\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.082733 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.082698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-web-config\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.085035 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.085000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.110196 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.110173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcw2\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-kube-api-access-shcw2\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.117187 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.116725    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v6s4d" podStartSLOduration=2.2820524779999998 podStartE2EDuration="8.116709964s" totalImagesPullingTime="5.834657486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:09.116576363 +0000 UTC m=+62.017720754" watchObservedRunningTime="2026-07-28 12:36:09.116709964 +0000 UTC m=+62.017854355"
Jul 28 12:36:09.377424 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.377393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:09.380123 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.380098    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b7d56b-f47c-48a5-b2b9-b26451aee178-node-exporter-tls\") pod \"node-exporter-l6shh\" (UID: \"51b7d56b-f47c-48a5-b2b9-b26451aee178\") " pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:09.506876 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.506688    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l6shh"
Jul 28 12:36:09.579804 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.579768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.579925 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.579839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.580837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.580812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.583837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.583804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.845900 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.845865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:36:09.890003 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.889957    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9s78"
Jul 28 12:36:09.974048 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.973961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw" event={"ID":"e679bdb4-1b51-4f64-b3b3-2567f68d030d","Type":"ContainerStarted","Data":"f7ed66c377ffd0709bf4e0baf0908268d9cd1f1c8ae8771d70d9b89e7daecc0c"}
Jul 28 12:36:09.976391 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:09.976343    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6shh" event={"ID":"51b7d56b-f47c-48a5-b2b9-b26451aee178","Type":"ContainerStarted","Data":"0aa58e7782f1bf5405656524eb06db7083ae10f3b9f9cc77606f44ebf0793f81"}
Jul 28 12:36:10.049665 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.049485    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:36:10.052525 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:10.052487    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5784be4c_eebd_4ee5_83dd_fcf383439637.slice/crio-3eeb75946034a49cabea34d65eb12b728f5067ad23d63c87cab391509530e1ba WatchSource:0}: Error finding container 3eeb75946034a49cabea34d65eb12b728f5067ad23d63c87cab391509530e1ba: Status 404 returned error can't find the container with id 3eeb75946034a49cabea34d65eb12b728f5067ad23d63c87cab391509530e1ba
Jul 28 12:36:10.836499 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.836471    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5996d84469-snzft"]
Jul 28 12:36:10.841356 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.840137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.844713 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.844690    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3f50di9mrsamv\""
Jul 28 12:36:10.845132 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.845091    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 12:36:10.845132 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.845121    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 12:36:10.845306 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.845138    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 12:36:10.845306 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.845201    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 12:36:10.845306 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.845205    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pdkwq\""
Jul 28 12:36:10.845952 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.845937    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 12:36:10.858885 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.858865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5996d84469-snzft"]
Jul 28 12:36:10.890979 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.890956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2w98\" (UniqueName: \"kubernetes.io/projected/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-kube-api-access-l2w98\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891090 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.890997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-tls\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891090 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.891027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-grpc-tls\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891090 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.891060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891239 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.891089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891239 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.891131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891239 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.891150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-metrics-client-ca\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.891239 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.891218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.980612 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.980584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"3eeb75946034a49cabea34d65eb12b728f5067ad23d63c87cab391509530e1ba"}
Jul 28 12:36:10.991975 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.991945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-tls\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992102 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.991995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-grpc-tls\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992102 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.992042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992102 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.992075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.992143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.992171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-metrics-client-ca\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.992206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.992263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.992260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2w98\" (UniqueName: \"kubernetes.io/projected/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-kube-api-access-l2w98\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.995223 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.995175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-tls\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.995329 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.995219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-metrics-client-ca\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.995421 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.995338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.995540 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.995508    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.996073 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.996032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.996226 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.996202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-grpc-tls\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:10.996401 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:10.996361    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:11.013796 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:11.013762    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2w98\" (UniqueName: \"kubernetes.io/projected/0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc-kube-api-access-l2w98\") pod \"thanos-querier-5996d84469-snzft\" (UID: \"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc\") " pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:11.151741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:11.151665    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:12.143499 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.143462    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-56dc799bb-l8mmf"]
Jul 28 12:36:12.165189 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.165160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56dc799bb-l8mmf"]
Jul 28 12:36:12.165346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.165255    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.169948 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.169885    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 12:36:12.169948 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.169915    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 12:36:12.169948 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.169885    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hld5p\""
Jul 28 12:36:12.169948 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.169921    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 12:36:12.170190 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.169983    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 12:36:12.170190 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.169920    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-em2196i088lpt\""
Jul 28 12:36:12.204377 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-secret-metrics-server-client-certs\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.204482 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcce6beb-54b5-46ee-b8de-e6de93704f43-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.204482 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-client-ca-bundle\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.204600 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpbs\" (UniqueName: \"kubernetes.io/projected/bcce6beb-54b5-46ee-b8de-e6de93704f43-kube-api-access-cwpbs\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.204600 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-secret-metrics-server-tls\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.204682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcce6beb-54b5-46ee-b8de-e6de93704f43-audit-log\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.204730 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.204699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcce6beb-54b5-46ee-b8de-e6de93704f43-metrics-server-audit-profiles\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.305784 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.305762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-secret-metrics-server-client-certs\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.305916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.305801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcce6beb-54b5-46ee-b8de-e6de93704f43-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.305916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.305822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-client-ca-bundle\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.305916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.305840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpbs\" (UniqueName: \"kubernetes.io/projected/bcce6beb-54b5-46ee-b8de-e6de93704f43-kube-api-access-cwpbs\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.305916 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.305875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-secret-metrics-server-tls\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.306109 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.305930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcce6beb-54b5-46ee-b8de-e6de93704f43-audit-log\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.306314 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.306281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcce6beb-54b5-46ee-b8de-e6de93704f43-audit-log\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.306314 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.306291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcce6beb-54b5-46ee-b8de-e6de93704f43-metrics-server-audit-profiles\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.306676 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.306654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcce6beb-54b5-46ee-b8de-e6de93704f43-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.307222 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.307200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcce6beb-54b5-46ee-b8de-e6de93704f43-metrics-server-audit-profiles\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.308421 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.308397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-secret-metrics-server-tls\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.308767 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.308746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-client-ca-bundle\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.308895 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.308853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcce6beb-54b5-46ee-b8de-e6de93704f43-secret-metrics-server-client-certs\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.316075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.316058    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpbs\" (UniqueName: \"kubernetes.io/projected/bcce6beb-54b5-46ee-b8de-e6de93704f43-kube-api-access-cwpbs\") pod \"metrics-server-56dc799bb-l8mmf\" (UID: \"bcce6beb-54b5-46ee-b8de-e6de93704f43\") " pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.475839 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.475752    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:12.488886 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.488851    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f98d57d6-zb7xk"]
Jul 28 12:36:12.490072 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.490049    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"]
Jul 28 12:36:12.493362 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.493344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.496303 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.496284    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 12:36:12.496667 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.496651    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6p774\""
Jul 28 12:36:12.505243 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.505220    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"]
Jul 28 12:36:12.526624 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.526599    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95c7c8b48-2xnx4"]
Jul 28 12:36:12.530005 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.529987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.545793 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.545770    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95c7c8b48-2xnx4"]
Jul 28 12:36:12.608746 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-trusted-ca-bundle\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.608859 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bmww\" (UniqueName: \"kubernetes.io/projected/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-kube-api-access-9bmww\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.608859 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-oauth-serving-cert\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.608859 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-serving-cert\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.609015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-config\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.609015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8dc012f0-ae6b-4f69-aedb-59860abacf47-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tmjl\" (UID: \"8dc012f0-ae6b-4f69-aedb-59860abacf47\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.609015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.608967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-oauth-config\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.609015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.609005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-service-ca\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.609191 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.609035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8trq\" (UniqueName: \"kubernetes.io/projected/8dc012f0-ae6b-4f69-aedb-59860abacf47-kube-api-access-w8trq\") pod \"monitoring-plugin-68b7fc9574-7tmjl\" (UID: \"8dc012f0-ae6b-4f69-aedb-59860abacf47\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.709500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-trusted-ca-bundle\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bmww\" (UniqueName: \"kubernetes.io/projected/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-kube-api-access-9bmww\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-oauth-serving-cert\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-serving-cert\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-config\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8dc012f0-ae6b-4f69-aedb-59860abacf47-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tmjl\" (UID: \"8dc012f0-ae6b-4f69-aedb-59860abacf47\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.709902 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-oauth-config\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709902 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-service-ca\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.709902 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.709761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8trq\" (UniqueName: \"kubernetes.io/projected/8dc012f0-ae6b-4f69-aedb-59860abacf47-kube-api-access-w8trq\") pod \"monitoring-plugin-68b7fc9574-7tmjl\" (UID: \"8dc012f0-ae6b-4f69-aedb-59860abacf47\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.710524 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.710497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-trusted-ca-bundle\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.711251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.710864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-service-ca\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.711251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.711215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-oauth-serving-cert\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.711977 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.711953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-config\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.713138 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.713093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-serving-cert\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.713236 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.713200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-oauth-config\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.713517 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.713497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8dc012f0-ae6b-4f69-aedb-59860abacf47-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7tmjl\" (UID: \"8dc012f0-ae6b-4f69-aedb-59860abacf47\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.720007 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.719960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8trq\" (UniqueName: \"kubernetes.io/projected/8dc012f0-ae6b-4f69-aedb-59860abacf47-kube-api-access-w8trq\") pod \"monitoring-plugin-68b7fc9574-7tmjl\" (UID: \"8dc012f0-ae6b-4f69-aedb-59860abacf47\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.720465 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.720445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bmww\" (UniqueName: \"kubernetes.io/projected/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-kube-api-access-9bmww\") pod \"console-95c7c8b48-2xnx4\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") " pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:12.804490 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.804428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:12.840453 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:12.840426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:13.416814 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.416781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:36:13.417242 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.416836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:36:13.420667 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.420645    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:36:13.421601 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.421588    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:36:13.430132 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.430113    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c80ac7ce-b3d4-481e-8c0f-6486a5a44827-metrics-certs\") pod \"network-metrics-daemon-hcvxx\" (UID: \"c80ac7ce-b3d4-481e-8c0f-6486a5a44827\") " pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:36:13.430323 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.430305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5352b7da-d4d1-4cca-9d6e-d522d0495f42-original-pull-secret\") pod \"global-pull-secret-syncer-rg6kq\" (UID: \"5352b7da-d4d1-4cca-9d6e-d522d0495f42\") " pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:36:13.440402 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.440358    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mf2p5\""
Jul 28 12:36:13.447196 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.447175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hcvxx"
Jul 28 12:36:13.518215 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.518188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:36:13.521811 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.521787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:36:13.531715 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.531694    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:36:13.536236 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.536220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg6kq"
Jul 28 12:36:13.541721 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.541703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rq9\" (UniqueName: \"kubernetes.io/projected/ce31f5aa-2e18-4923-8c01-a630b25d1e52-kube-api-access-n5rq9\") pod \"network-check-target-h9c6h\" (UID: \"ce31f5aa-2e18-4923-8c01-a630b25d1e52\") " pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:36:13.761882 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.761813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cc56m\""
Jul 28 12:36:13.769284 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:13.769261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:36:14.374002 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.373963    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:14.377611 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.377592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.385246 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.385230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 12:36:14.385510 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.385494    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 12:36:14.386086 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.386066    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 12:36:14.386086 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.386080    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-34b9ptsam9mqu\""
Jul 28 12:36:14.386258 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.386102    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 12:36:14.386642 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.386623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 12:36:14.387031 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387009    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 12:36:14.387214 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387200    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 12:36:14.387415 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387399    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 12:36:14.387797 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387774    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 12:36:14.387876 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387807    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 12:36:14.387876 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387786    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 12:36:14.387876 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387782    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-54m2v\""
Jul 28 12:36:14.388016 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.387784    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 12:36:14.389633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.389614    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 12:36:14.392887 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.392866    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 12:36:14.420320 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.420303    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.424980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.425682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.426553 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrdq\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-kube-api-access-8qrdq\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.426553 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-config\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.426553 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.426553 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.425749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533889    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrdq\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-kube-api-access-8qrdq\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-config\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.533991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.534484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.535424 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.535424 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.535424 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.534262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.545726 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.536269    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.545726 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.536871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.545726 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.539094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.545726 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.543876    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.545726 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.544553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.545726 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.545081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.553807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.546536    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.553807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.547159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.553807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.553668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.555040 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.554992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-config\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.561728 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.561681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrdq\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-kube-api-access-8qrdq\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.571550 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.571502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.572156 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.572115    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.572718 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.572680    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.573180 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.573140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.573826 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.573787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.574435 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.574383    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.576047 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.575977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.578444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.577783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.581337 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.581286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.635679 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.635564    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5996d84469-snzft"]
Jul 28 12:36:14.689896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.689837    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:14.817884 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.817787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hcvxx"]
Jul 28 12:36:14.910778 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.910680    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:36:14.915055 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.914439    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h9c6h"]
Jul 28 12:36:14.915272 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:14.915247    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c034273_5972_4ca0_a34d_41753e620f98.slice/crio-8bbe38b00cc8a917249b962c8c907ad32ba077bbd917bc85d1073aa34d67b6ab WatchSource:0}: Error finding container 8bbe38b00cc8a917249b962c8c907ad32ba077bbd917bc85d1073aa34d67b6ab: Status 404 returned error can't find the container with id 8bbe38b00cc8a917249b962c8c907ad32ba077bbd917bc85d1073aa34d67b6ab
Jul 28 12:36:14.916959 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:14.916937    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce31f5aa_2e18_4923_8c01_a630b25d1e52.slice/crio-42562afcce7e5970c5d1c618fb6223e53ef89c702a521d468398f4246c194a24 WatchSource:0}: Error finding container 42562afcce7e5970c5d1c618fb6223e53ef89c702a521d468398f4246c194a24: Status 404 returned error can't find the container with id 42562afcce7e5970c5d1c618fb6223e53ef89c702a521d468398f4246c194a24
Jul 28 12:36:14.930504 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.929866    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-56dc799bb-l8mmf"]
Jul 28 12:36:14.935909 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.935851    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95c7c8b48-2xnx4"]
Jul 28 12:36:14.937251 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:14.937227    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcce6beb_54b5_46ee_b8de_e6de93704f43.slice/crio-2accf38493bb531befa1d0991f14343ac08bc9a40d34c852499b4a4f23cefbfa WatchSource:0}: Error finding container 2accf38493bb531befa1d0991f14343ac08bc9a40d34c852499b4a4f23cefbfa: Status 404 returned error can't find the container with id 2accf38493bb531befa1d0991f14343ac08bc9a40d34c852499b4a4f23cefbfa
Jul 28 12:36:14.938234 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:14.938214    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28e3dff_fffa_4a9e_bcc1_36efdccc0fad.slice/crio-cc9fff7a5476db4a2f7e354e24dd07b9f74b3cfe4a29c3c3511d91ef83228022 WatchSource:0}: Error finding container cc9fff7a5476db4a2f7e354e24dd07b9f74b3cfe4a29c3c3511d91ef83228022: Status 404 returned error can't find the container with id cc9fff7a5476db4a2f7e354e24dd07b9f74b3cfe4a29c3c3511d91ef83228022
Jul 28 12:36:14.943676 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.943511    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rg6kq"]
Jul 28 12:36:14.944639 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:14.944585    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"]
Jul 28 12:36:14.946860 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:14.946844    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5352b7da_d4d1_4cca_9d6e_d522d0495f42.slice/crio-15e93f507e341f3fea039131e470e992bd9e8e9f47d9d86468c7a5cfd12f6e2d WatchSource:0}: Error finding container 15e93f507e341f3fea039131e470e992bd9e8e9f47d9d86468c7a5cfd12f6e2d: Status 404 returned error can't find the container with id 15e93f507e341f3fea039131e470e992bd9e8e9f47d9d86468c7a5cfd12f6e2d
Jul 28 12:36:14.948936 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:36:14.948913    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc012f0_ae6b_4f69_aedb_59860abacf47.slice/crio-050651a8ee2330c3af29964e31d805ae59dba9684030e87df90b0a5541969973 WatchSource:0}: Error finding container 050651a8ee2330c3af29964e31d805ae59dba9684030e87df90b0a5541969973: Status 404 returned error can't find the container with id 050651a8ee2330c3af29964e31d805ae59dba9684030e87df90b0a5541969973
Jul 28 12:36:15.003922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.003817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f" event={"ID":"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec","Type":"ContainerStarted","Data":"77c4203a29d5639c52e81e1ed2291294d5684e650fd09b7aeb5f43d7078b3629"}
Jul 28 12:36:15.003922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.003895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f" event={"ID":"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec","Type":"ContainerStarted","Data":"06758fb8677471f64dd6c62ab269a07c5941493256b22cc330a7426e17ae597e"}
Jul 28 12:36:15.004092 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.003946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f" event={"ID":"5c2f3458-2fa4-49b5-9c3e-691b1e6276ec","Type":"ContainerStarted","Data":"2010b4ebf689bdb4423255bc1c208557c28a84a1427917644d20ffb8485f265e"}
Jul 28 12:36:15.007338 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.007245    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f98d57d6-zb7xk" event={"ID":"cd61ac96-302b-454d-a9c1-d53aa35309fa","Type":"ContainerStarted","Data":"e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6"}
Jul 28 12:36:15.010137 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.010077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7c8b48-2xnx4" event={"ID":"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad","Type":"ContainerStarted","Data":"cc9fff7a5476db4a2f7e354e24dd07b9f74b3cfe4a29c3c3511d91ef83228022"}
Jul 28 12:36:15.011395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.011208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl" event={"ID":"8dc012f0-ae6b-4f69-aedb-59860abacf47","Type":"ContainerStarted","Data":"050651a8ee2330c3af29964e31d805ae59dba9684030e87df90b0a5541969973"}
Jul 28 12:36:15.013614 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.013588    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5" exitCode=0
Jul 28 12:36:15.013731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.013669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"}
Jul 28 12:36:15.015915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.015869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rg6kq" event={"ID":"5352b7da-d4d1-4cca-9d6e-d522d0495f42","Type":"ContainerStarted","Data":"15e93f507e341f3fea039131e470e992bd9e8e9f47d9d86468c7a5cfd12f6e2d"}
Jul 28 12:36:15.018119 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.018067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf94z" event={"ID":"87172efd-f000-4b9c-8019-f1e930b2a804","Type":"ContainerStarted","Data":"1b3933968de923afe827428e67ab2b01262bfde54b8103339ae8dea4351254f3"}
Jul 28 12:36:15.021125 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.021103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw" event={"ID":"e679bdb4-1b51-4f64-b3b3-2567f68d030d","Type":"ContainerStarted","Data":"29ba4c9f234a99d98f2c9b884d8842dd8a7fc9b7abb321873fef23547f66b3ed"}
Jul 28 12:36:15.023571 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.023548    2576 generic.go:356] "Generic (PLEG): container finished" podID="51b7d56b-f47c-48a5-b2b9-b26451aee178" containerID="979ccb006251c48cf728a02cf76bae990f57c307cbbfbd510f8fef03d60b13a9" exitCode=0
Jul 28 12:36:15.023645 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.023616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6shh" event={"ID":"51b7d56b-f47c-48a5-b2b9-b26451aee178","Type":"ContainerDied","Data":"979ccb006251c48cf728a02cf76bae990f57c307cbbfbd510f8fef03d60b13a9"}
Jul 28 12:36:15.027326 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.027305    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"}
Jul 28 12:36:15.027444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.027419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"8bbe38b00cc8a917249b962c8c907ad32ba077bbd917bc85d1073aa34d67b6ab"}
Jul 28 12:36:15.031409 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.031348    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-rpv9f" podStartSLOduration=2.834294852 podStartE2EDuration="8.031331269s" totalImagesPullingTime="5.197036417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:15.02900013 +0000 UTC m=+67.930144521" watchObservedRunningTime="2026-07-28 12:36:15.031331269 +0000 UTC m=+67.932475660"
Jul 28 12:36:15.031512 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.031454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hcvxx" event={"ID":"c80ac7ce-b3d4-481e-8c0f-6486a5a44827","Type":"ContainerStarted","Data":"322f3a5f054402fb36e1fd6f9baa057d08e264735fb52858b997a06710261687"}
Jul 28 12:36:15.032790 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.032767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h9c6h" event={"ID":"ce31f5aa-2e18-4923-8c01-a630b25d1e52","Type":"ContainerStarted","Data":"42562afcce7e5970c5d1c618fb6223e53ef89c702a521d468398f4246c194a24"}
Jul 28 12:36:15.033925 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.033907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf" event={"ID":"bcce6beb-54b5-46ee-b8de-e6de93704f43","Type":"ContainerStarted","Data":"2accf38493bb531befa1d0991f14343ac08bc9a40d34c852499b4a4f23cefbfa"}
Jul 28 12:36:15.035477 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.035365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"e380f08699096614dc15b56c151c32406bef6c74fa5b04b0b6925f4045b7612e"}
Jul 28 12:36:15.094342 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.094280    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85f98d57d6-zb7xk" podStartSLOduration=2.712278527 podStartE2EDuration="9.094264318s" totalImagesPullingTime="6.381985791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:15.092886806 +0000 UTC m=+67.994031186" watchObservedRunningTime="2026-07-28 12:36:15.094264318 +0000 UTC m=+67.995408685"
Jul 28 12:36:15.146712 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:15.146669    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6r5gw" podStartSLOduration=3.109748803 podStartE2EDuration="8.146654141s" totalImagesPullingTime="5.036905338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:15.14573254 +0000 UTC m=+68.046876931" watchObservedRunningTime="2026-07-28 12:36:15.146654141 +0000 UTC m=+68.047798509"
Jul 28 12:36:16.046692 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.046652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6shh" event={"ID":"51b7d56b-f47c-48a5-b2b9-b26451aee178","Type":"ContainerStarted","Data":"91534ef5f36d5d41f4b4aa76fb80f6915600e0cd0945d48720722d76beb2afe4"}
Jul 28 12:36:16.047150 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.046756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l6shh" event={"ID":"51b7d56b-f47c-48a5-b2b9-b26451aee178","Type":"ContainerStarted","Data":"af36c2dadb373802349d414bf9e6c08a188e857ca2fa56156472b484fea11cf4"}
Jul 28 12:36:16.051942 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.050589    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93" exitCode=0
Jul 28 12:36:16.051942 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.051865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"}
Jul 28 12:36:16.061156 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.061084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7c8b48-2xnx4" event={"ID":"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad","Type":"ContainerStarted","Data":"7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107"}
Jul 28 12:36:16.076656 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.076228    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l6shh" podStartSLOduration=4.201508539 podStartE2EDuration="9.076209678s" totalImagesPullingTime="4.751139223s" totalInitContainerRuntime="123.561916ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:16.074639188 +0000 UTC m=+68.975783577" watchObservedRunningTime="2026-07-28 12:36:16.076209678 +0000 UTC m=+68.977354069"
Jul 28 12:36:16.099390 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.098785    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95c7c8b48-2xnx4" podStartSLOduration=4.098767437 podStartE2EDuration="4.098767437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:16.096718502 +0000 UTC m=+68.997862890" watchObservedRunningTime="2026-07-28 12:36:16.098767437 +0000 UTC m=+68.999911829"
Jul 28 12:36:16.553117 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:16.553084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:17.073039 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:17.072524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hcvxx" event={"ID":"c80ac7ce-b3d4-481e-8c0f-6486a5a44827","Type":"ContainerStarted","Data":"626b496b423a74f875ab8513817d805f08a02c36ecbfc69f0fce691965f08959"}
Jul 28 12:36:17.073039 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:17.072577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hcvxx" event={"ID":"c80ac7ce-b3d4-481e-8c0f-6486a5a44827","Type":"ContainerStarted","Data":"419028cb4c2f406df7377bae8f8b1eefdc607cb434215e1bc3945eb4d220b0dc"}
Jul 28 12:36:18.979322 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:18.979287    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-s9nqg"
Jul 28 12:36:19.001259 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:19.001169    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hcvxx" podStartSLOduration=69.732674108 podStartE2EDuration="1m11.001150073s" totalImagesPullingTime="1.268475965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:17.094652717 +0000 UTC m=+69.995797109" watchObservedRunningTime="2026-07-28 12:36:19.001150073 +0000 UTC m=+71.902294466"
Jul 28 12:36:22.841276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:22.841243    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:22.841728 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:22.841329    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:22.848276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:22.848100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:23.103118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:23.103042    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:36:25.107001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.106919    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl" event={"ID":"8dc012f0-ae6b-4f69-aedb-59860abacf47","Type":"ContainerStarted","Data":"edcfe36393a0829eb6818792808dc456cee24349d0495eda04ab2f1421f0fc8c"}
Jul 28 12:36:25.107406 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.107151    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:25.110660 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.110632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"}
Jul 28 12:36:25.110779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.110701    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"}
Jul 28 12:36:25.110779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.110740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"}
Jul 28 12:36:25.110779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.110777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"}
Jul 28 12:36:25.110932 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.110820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"}
Jul 28 12:36:25.112528 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.112495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rg6kq" event={"ID":"5352b7da-d4d1-4cca-9d6e-d522d0495f42","Type":"ContainerStarted","Data":"00ee0ceff7a8d25d647238f217dcee64ce65c5e8e914efc3a3b3eb1f7c42ecdc"}
Jul 28 12:36:25.113111 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.113093    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl"
Jul 28 12:36:25.114663 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.114640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf94z" event={"ID":"87172efd-f000-4b9c-8019-f1e930b2a804","Type":"ContainerStarted","Data":"e7fbc6b38060a5b89c9b5d719857544053cb893c5110f627ac0b110151685a50"}
Jul 28 12:36:25.118075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.118057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"}
Jul 28 12:36:25.118160 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.118107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"}
Jul 28 12:36:25.118160 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.118145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"}
Jul 28 12:36:25.118260 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.118184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"}
Jul 28 12:36:25.118260 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.118224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"}
Jul 28 12:36:25.118260 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.118258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerStarted","Data":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"}
Jul 28 12:36:25.119592 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.119573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h9c6h" event={"ID":"ce31f5aa-2e18-4923-8c01-a630b25d1e52","Type":"ContainerStarted","Data":"24428ead80897f51914bd518166c92764eeb13c46e90ad9738f1653b77a845de"}
Jul 28 12:36:25.119719 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.119690    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:36:25.121061 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.121042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf" event={"ID":"bcce6beb-54b5-46ee-b8de-e6de93704f43","Type":"ContainerStarted","Data":"a9b0425edfabb57aff02decc583e70afee8698961209a687251d9fe0a47b2170"}
Jul 28 12:36:25.122999 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.122982    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"98230752823ac289522794f9286907fb72331ccec94fc0b73e380d6df0ee5e86"}
Jul 28 12:36:25.123081 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.123028    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"b5c124ae113069b87ee4ddd822d17cda7871cb9ace3e4d5f4e5d0dbd12f50df1"}
Jul 28 12:36:25.123081 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.123052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"0defaa6f66caf2093d99f1ef06817badc4cb3fd6eb9780636d2802bedb75de9e"}
Jul 28 12:36:25.133723 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.133681    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7tmjl" podStartSLOduration=3.7498770649999997 podStartE2EDuration="13.133670471s" totalImagesPullingTime="9.383793406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.130430993 +0000 UTC m=+78.031575382" watchObservedRunningTime="2026-07-28 12:36:25.133670471 +0000 UTC m=+78.034814852"
Jul 28 12:36:25.159191 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.159151    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf" podStartSLOduration=3.70812988 podStartE2EDuration="13.159138691s" totalImagesPullingTime="9.451008811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.158130468 +0000 UTC m=+78.059274861" watchObservedRunningTime="2026-07-28 12:36:25.159138691 +0000 UTC m=+78.060283080"
Jul 28 12:36:25.198961 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.198924    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.821132165 podStartE2EDuration="11.198912658s" totalImagesPullingTime="8.353517785s" totalInitContainerRuntime="24.262708ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.198720797 +0000 UTC m=+78.099865187" watchObservedRunningTime="2026-07-28 12:36:25.198912658 +0000 UTC m=+78.100057048"
Jul 28 12:36:25.229967 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.229920    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kf94z" podStartSLOduration=2.167572749 podStartE2EDuration="24.229907608s" totalImagesPullingTime="22.062334859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:36:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.229169628 +0000 UTC m=+78.130314019" watchObservedRunningTime="2026-07-28 12:36:25.229907608 +0000 UTC m=+78.131051998"
Jul 28 12:36:25.289537 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:25.289496    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rg6kq" podStartSLOduration=67.828415485 podStartE2EDuration="1m17.289481032s" totalImagesPullingTime="9.461065547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.288793925 +0000 UTC m=+78.189938315" watchObservedRunningTime="2026-07-28 12:36:25.289481032 +0000 UTC m=+78.190625422"
Jul 28 12:36:26.130958 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.130930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerStarted","Data":"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"}
Jul 28 12:36:26.133896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.133869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"8fea71bccd4171a59dd028be735c04743aeea4b4f33785d65a722e7c7bf15fb7"}
Jul 28 12:36:26.134009 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.133938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"5f2cfe66c6d5285d15386c7fa556e3673deb0d31d036e0c53fa7a2080bc8e722"}
Jul 28 12:36:26.134009 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.133967    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" event={"ID":"0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc","Type":"ContainerStarted","Data":"e03085e4488aeff3bf3bd33df494919bd82bda1ef11b295a90f9e5c96cbaa4ce"}
Jul 28 12:36:26.178096 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.178060    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.686167256 podStartE2EDuration="18.178047442s" totalImagesPullingTime="14.457211579s" totalInitContainerRuntime="34.668607ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:26.17234227 +0000 UTC m=+79.073486659" watchObservedRunningTime="2026-07-28 12:36:26.178047442 +0000 UTC m=+79.079191826"
Jul 28 12:36:26.178458 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.178436    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h9c6h" podStartSLOduration=68.707538418 podStartE2EDuration="1m18.178428503s" totalImagesPullingTime="9.470890085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:25.329636803 +0000 UTC m=+78.230781194" watchObservedRunningTime="2026-07-28 12:36:26.178428503 +0000 UTC m=+79.079572893"
Jul 28 12:36:26.263595 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:26.263548    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5996d84469-snzft" podStartSLOduration=5.711650995 podStartE2EDuration="16.263535168s" totalImagesPullingTime="10.551884173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:36:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:36:26.263521301 +0000 UTC m=+79.164665691" watchObservedRunningTime="2026-07-28 12:36:26.263535168 +0000 UTC m=+79.164679559"
Jul 28 12:36:27.138425 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:27.138385    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:29.691147 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:29.690980    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:36:32.476641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:32.476601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:32.477113 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:32.476699    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:33.146814 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:33.146791    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5996d84469-snzft"
Jul 28 12:36:40.030956 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.030900    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85f98d57d6-zb7xk" podUID="cd61ac96-302b-454d-a9c1-d53aa35309fa" containerName="console" containerID="cri-o://e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6" gracePeriod=15
Jul 28 12:36:40.803875 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.803852    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:40.866200 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866172    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwq8f\" (UniqueName: \"kubernetes.io/projected/cd61ac96-302b-454d-a9c1-d53aa35309fa-kube-api-access-dwq8f\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866309 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866233    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-trusted-ca-bundle\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866309 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866263    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-config\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866402 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866321    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-serving-cert\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866402 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866353    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-oauth-config\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866402 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-service-ca\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866544 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-oauth-serving-cert\") pod \"cd61ac96-302b-454d-a9c1-d53aa35309fa\" (UID: \"cd61ac96-302b-454d-a9c1-d53aa35309fa\") "
Jul 28 12:36:40.866837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-config" (OuterVolumeSpecName: "console-config") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:40.866837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866805    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:40.866837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866820    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:40.867025 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.866926    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:36:40.868567 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.868544    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:40.868658 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.868604    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd61ac96-302b-454d-a9c1-d53aa35309fa-kube-api-access-dwq8f" (OuterVolumeSpecName: "kube-api-access-dwq8f") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "kube-api-access-dwq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:36:40.868658 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.868634    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cd61ac96-302b-454d-a9c1-d53aa35309fa" (UID: "cd61ac96-302b-454d-a9c1-d53aa35309fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:36:40.967230 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967206    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:40.967230 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967228    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-oauth-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:40.967350 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967237    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-service-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:40.967350 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967246    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-oauth-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:40.967350 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967254    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwq8f\" (UniqueName: \"kubernetes.io/projected/cd61ac96-302b-454d-a9c1-d53aa35309fa-kube-api-access-dwq8f\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:40.967350 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967263    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:40.967350 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:40.967273    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd61ac96-302b-454d-a9c1-d53aa35309fa-console-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:36:41.194827 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.194796    2576 generic.go:356] "Generic (PLEG): container finished" podID="cd61ac96-302b-454d-a9c1-d53aa35309fa" containerID="e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6" exitCode=0
Jul 28 12:36:41.195184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.194851    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f98d57d6-zb7xk"
Jul 28 12:36:41.195184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.194894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f98d57d6-zb7xk" event={"ID":"cd61ac96-302b-454d-a9c1-d53aa35309fa","Type":"ContainerDied","Data":"e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6"}
Jul 28 12:36:41.195184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.194953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f98d57d6-zb7xk" event={"ID":"cd61ac96-302b-454d-a9c1-d53aa35309fa","Type":"ContainerDied","Data":"d8f55775c6db12d800d94b3a7492e4f7daab120b05d653d46621e610db8d5894"}
Jul 28 12:36:41.195184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.194976    2576 scope.go:117] "RemoveContainer" containerID="e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6"
Jul 28 12:36:41.205044 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.205026    2576 scope.go:117] "RemoveContainer" containerID="e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6"
Jul 28 12:36:41.205351 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:36:41.205328    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6\": container with ID starting with e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6 not found: ID does not exist" containerID="e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6"
Jul 28 12:36:41.205467 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.205359    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6"} err="failed to get container status \"e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6\": rpc error: code = NotFound desc = could not find container \"e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6\": container with ID starting with e8d9e81624c03f6f446723b7769ad90ada6b81f1b6ccbf3c931c1342af8832b6 not found: ID does not exist"
Jul 28 12:36:41.219947 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.219924    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f98d57d6-zb7xk"]
Jul 28 12:36:41.223724 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.223701    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85f98d57d6-zb7xk"]
Jul 28 12:36:41.723983 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:41.723955    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd61ac96-302b-454d-a9c1-d53aa35309fa" path="/var/lib/kubelet/pods/cd61ac96-302b-454d-a9c1-d53aa35309fa/volumes"
Jul 28 12:36:52.481749 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:52.481722    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:52.490045 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:52.490021    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-56dc799bb-l8mmf"
Jul 28 12:36:56.137225 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:36:56.137198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h9c6h"
Jul 28 12:37:14.690574 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:14.690534    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:14.710910 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:14.710883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:15.343423 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:15.343393    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:28.402666 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.402623    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:37:28.403151 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.403090    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="alertmanager" containerID="cri-o://45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a" gracePeriod=120
Jul 28 12:37:28.403151 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.403109    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-metric" containerID="cri-o://1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d" gracePeriod=120
Jul 28 12:37:28.403258 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.403172    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="prom-label-proxy" containerID="cri-o://a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04" gracePeriod=120
Jul 28 12:37:28.403304 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.403183    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-web" containerID="cri-o://c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f" gracePeriod=120
Jul 28 12:37:28.403883 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.403417    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy" containerID="cri-o://7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c" gracePeriod=120
Jul 28 12:37:28.403883 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:28.403428    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="config-reloader" containerID="cri-o://353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1" gracePeriod=120
Jul 28 12:37:29.384797 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384768    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04" exitCode=0
Jul 28 12:37:29.384797 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384792    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c" exitCode=0
Jul 28 12:37:29.384797 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384798    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1" exitCode=0
Jul 28 12:37:29.385011 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384804    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a" exitCode=0
Jul 28 12:37:29.385011 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"}
Jul 28 12:37:29.385011 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384923    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"}
Jul 28 12:37:29.385011 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"}
Jul 28 12:37:29.385011 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.384978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"}
Jul 28 12:37:29.635513 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.635464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:29.713991 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.713961    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.713996    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-cluster-tls-config\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-web\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714055    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714078    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shcw2\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-kube-api-access-shcw2\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-web-config\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714141    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714168    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-config-volume\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714196    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-tls-assets\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714252    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714298    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-main-db\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-config-out\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714351    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:29.714719 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714412    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-metrics-client-ca\") pod \"5784be4c-eebd-4ee5-83dd-fcf383439637\" (UID: \"5784be4c-eebd-4ee5-83dd-fcf383439637\") "
Jul 28 12:37:29.714719 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.714699    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.715169 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.715001    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:37:29.715843 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.715799    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:29.717049 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.717011    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.717790 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.717742    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.718075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.718038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.718630 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.718577    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-config-out" (OuterVolumeSpecName: "config-out") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:37:29.718839 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.718819    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-kube-api-access-shcw2" (OuterVolumeSpecName: "kube-api-access-shcw2") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "kube-api-access-shcw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:29.719901 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.719878    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:29.720225 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.720176    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-config-volume" (OuterVolumeSpecName: "config-volume") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.720441 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.720334    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.726258 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.726231    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.730481 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.730445    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-web-config" (OuterVolumeSpecName: "web-config") pod "5784be4c-eebd-4ee5-83dd-fcf383439637" (UID: "5784be4c-eebd-4ee5-83dd-fcf383439637"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:29.815519 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815496    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-cluster-tls-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815520    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815535    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-main-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815547    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shcw2\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-kube-api-access-shcw2\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815559    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-web-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815572    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815584    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-config-volume\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815595    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5784be4c-eebd-4ee5-83dd-fcf383439637-tls-assets\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815617 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815607    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5784be4c-eebd-4ee5-83dd-fcf383439637-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815879 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815621    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-alertmanager-main-db\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815879 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815633    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5784be4c-eebd-4ee5-83dd-fcf383439637-config-out\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:29.815879 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:29.815645    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5784be4c-eebd-4ee5-83dd-fcf383439637-metrics-client-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:30.392212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392178    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d" exitCode=0
Jul 28 12:37:30.392212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392204    2576 generic.go:356] "Generic (PLEG): container finished" podID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerID="c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f" exitCode=0
Jul 28 12:37:30.392441 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"}
Jul 28 12:37:30.392441 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392290    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.392441 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392343    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"}
Jul 28 12:37:30.392441 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5784be4c-eebd-4ee5-83dd-fcf383439637","Type":"ContainerDied","Data":"3eeb75946034a49cabea34d65eb12b728f5067ad23d63c87cab391509530e1ba"}
Jul 28 12:37:30.392441 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.392396    2576 scope.go:117] "RemoveContainer" containerID="a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"
Jul 28 12:37:30.400691 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.400665    2576 scope.go:117] "RemoveContainer" containerID="1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"
Jul 28 12:37:30.407638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.407620    2576 scope.go:117] "RemoveContainer" containerID="7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"
Jul 28 12:37:30.414627 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.414596    2576 scope.go:117] "RemoveContainer" containerID="c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"
Jul 28 12:37:30.417471 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.417450    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:37:30.420701 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.420677    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:37:30.422336 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.422321    2576 scope.go:117] "RemoveContainer" containerID="353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"
Jul 28 12:37:30.428961 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.428946    2576 scope.go:117] "RemoveContainer" containerID="45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"
Jul 28 12:37:30.435589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.435573    2576 scope.go:117] "RemoveContainer" containerID="ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"
Jul 28 12:37:30.442275 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.442258    2576 scope.go:117] "RemoveContainer" containerID="a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"
Jul 28 12:37:30.442553 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.442526    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04\": container with ID starting with a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04 not found: ID does not exist" containerID="a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"
Jul 28 12:37:30.442619 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.442560    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"} err="failed to get container status \"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04\": rpc error: code = NotFound desc = could not find container \"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04\": container with ID starting with a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04 not found: ID does not exist"
Jul 28 12:37:30.442619 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.442577    2576 scope.go:117] "RemoveContainer" containerID="1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"
Jul 28 12:37:30.442814 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.442794    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d\": container with ID starting with 1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d not found: ID does not exist" containerID="1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"
Jul 28 12:37:30.442859 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.442820    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"} err="failed to get container status \"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d\": rpc error: code = NotFound desc = could not find container \"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d\": container with ID starting with 1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d not found: ID does not exist"
Jul 28 12:37:30.442859 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.442834    2576 scope.go:117] "RemoveContainer" containerID="7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"
Jul 28 12:37:30.443016 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.443001    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c\": container with ID starting with 7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c not found: ID does not exist" containerID="7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"
Jul 28 12:37:30.443055 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443022    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"} err="failed to get container status \"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c\": rpc error: code = NotFound desc = could not find container \"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c\": container with ID starting with 7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c not found: ID does not exist"
Jul 28 12:37:30.443055 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443035    2576 scope.go:117] "RemoveContainer" containerID="c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"
Jul 28 12:37:30.443236 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.443222    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f\": container with ID starting with c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f not found: ID does not exist" containerID="c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"
Jul 28 12:37:30.443273 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443240    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"} err="failed to get container status \"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f\": rpc error: code = NotFound desc = could not find container \"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f\": container with ID starting with c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f not found: ID does not exist"
Jul 28 12:37:30.443273 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443254    2576 scope.go:117] "RemoveContainer" containerID="353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"
Jul 28 12:37:30.443442 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.443426    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1\": container with ID starting with 353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1 not found: ID does not exist" containerID="353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"
Jul 28 12:37:30.443489 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443445    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"} err="failed to get container status \"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1\": rpc error: code = NotFound desc = could not find container \"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1\": container with ID starting with 353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1 not found: ID does not exist"
Jul 28 12:37:30.443489 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443457    2576 scope.go:117] "RemoveContainer" containerID="45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"
Jul 28 12:37:30.443659 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.443640    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a\": container with ID starting with 45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a not found: ID does not exist" containerID="45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"
Jul 28 12:37:30.443724 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443669    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"} err="failed to get container status \"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a\": rpc error: code = NotFound desc = could not find container \"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a\": container with ID starting with 45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a not found: ID does not exist"
Jul 28 12:37:30.443724 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443690    2576 scope.go:117] "RemoveContainer" containerID="ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"
Jul 28 12:37:30.443917 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:30.443902    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5\": container with ID starting with ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5 not found: ID does not exist" containerID="ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"
Jul 28 12:37:30.443962 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443919    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"} err="failed to get container status \"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5\": rpc error: code = NotFound desc = could not find container \"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5\": container with ID starting with ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5 not found: ID does not exist"
Jul 28 12:37:30.443962 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.443930    2576 scope.go:117] "RemoveContainer" containerID="a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"
Jul 28 12:37:30.444109 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444095    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04"} err="failed to get container status \"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04\": rpc error: code = NotFound desc = could not find container \"a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04\": container with ID starting with a59dc289b051707436f55bb3a719330c8335927de10525d0e2a77de195899d04 not found: ID does not exist"
Jul 28 12:37:30.444109 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444109    2576 scope.go:117] "RemoveContainer" containerID="1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"
Jul 28 12:37:30.444280 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444267    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d"} err="failed to get container status \"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d\": rpc error: code = NotFound desc = could not find container \"1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d\": container with ID starting with 1c554c9fb2e3149b2588791d421ebfb799410e4cb5dc734502a40e7a42a38e4d not found: ID does not exist"
Jul 28 12:37:30.444280 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444279    2576 scope.go:117] "RemoveContainer" containerID="7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"
Jul 28 12:37:30.444496 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444477    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c"} err="failed to get container status \"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c\": rpc error: code = NotFound desc = could not find container \"7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c\": container with ID starting with 7eb7cc72ae7a722bac17cba565ef7306e0f859c3bccf8fe8fdb11baf6a97b39c not found: ID does not exist"
Jul 28 12:37:30.444575 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444498    2576 scope.go:117] "RemoveContainer" containerID="c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"
Jul 28 12:37:30.444730 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444713    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f"} err="failed to get container status \"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f\": rpc error: code = NotFound desc = could not find container \"c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f\": container with ID starting with c78b277fbfcf7ec793d4ecdb42db5d59cb6c3ba741303011b9920adee483787f not found: ID does not exist"
Jul 28 12:37:30.444799 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444732    2576 scope.go:117] "RemoveContainer" containerID="353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"
Jul 28 12:37:30.444960 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444942    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1"} err="failed to get container status \"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1\": rpc error: code = NotFound desc = could not find container \"353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1\": container with ID starting with 353ef752c64e4ba9124c7bc606331112715282ae24b58c0bdf756ce4657cd0b1 not found: ID does not exist"
Jul 28 12:37:30.445025 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.444962    2576 scope.go:117] "RemoveContainer" containerID="45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"
Jul 28 12:37:30.445181 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.445166    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a"} err="failed to get container status \"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a\": rpc error: code = NotFound desc = could not find container \"45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a\": container with ID starting with 45f6d959208b658d916491ee388347cdbe64915d17464967fd31a1adb250624a not found: ID does not exist"
Jul 28 12:37:30.445249 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.445182    2576 scope.go:117] "RemoveContainer" containerID="ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"
Jul 28 12:37:30.445455 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.445428    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5"} err="failed to get container status \"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5\": rpc error: code = NotFound desc = could not find container \"ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5\": container with ID starting with ae10b0e698f298feb506836a2a557a353c6b7843f1c49b176797eeb90dc424c5 not found: ID does not exist"
Jul 28 12:37:30.453765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.453744    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:37:30.454020 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454009    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd61ac96-302b-454d-a9c1-d53aa35309fa" containerName="console"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454022    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd61ac96-302b-454d-a9c1-d53aa35309fa" containerName="console"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454032    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="prom-label-proxy"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454038    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="prom-label-proxy"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454044    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-web"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454049    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-web"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454059    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="init-config-reloader"
Jul 28 12:37:30.454065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454064    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="init-config-reloader"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454076    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="alertmanager"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454081    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="alertmanager"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454089    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454094    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454100    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-metric"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454105    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-metric"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454111    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="config-reloader"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454116    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="config-reloader"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454176    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454184    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-metric"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454189    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="prom-label-proxy"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454194    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="config-reloader"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454201    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="alertmanager"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454206    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" containerName="kube-rbac-proxy-web"
Jul 28 12:37:30.454255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.454210    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd61ac96-302b-454d-a9c1-d53aa35309fa" containerName="console"
Jul 28 12:37:30.459271 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.459037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.461981 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.461957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 12:37:30.462074 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.461994    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 12:37:30.462074 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.461961    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 12:37:30.462175 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.462086    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 12:37:30.462175 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.462092    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dbs7f\""
Jul 28 12:37:30.462175 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.461957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 12:37:30.462175 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.462163    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 12:37:30.462359 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.462265    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 12:37:30.462359 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.462271    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 12:37:30.466812 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.466793    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 12:37:30.470828 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.470804    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:37:30.519539 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0540da8-e3fe-425f-9634-0791ba686651-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0540da8-e3fe-425f-9634-0791ba686651-config-out\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0540da8-e3fe-425f-9634-0791ba686651-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0540da8-e3fe-425f-9634-0791ba686651-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-web-config\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0540da8-e3fe-425f-9634-0791ba686651-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v482w\" (UniqueName: \"kubernetes.io/projected/e0540da8-e3fe-425f-9634-0791ba686651-kube-api-access-v482w\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519965 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519965 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.519965 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.519833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620493 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620574 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620574 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620574 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0540da8-e3fe-425f-9634-0791ba686651-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620574 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0540da8-e3fe-425f-9634-0791ba686651-config-out\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620778 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0540da8-e3fe-425f-9634-0791ba686651-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620778 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0540da8-e3fe-425f-9634-0791ba686651-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620778 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-web-config\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0540da8-e3fe-425f-9634-0791ba686651-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v482w\" (UniqueName: \"kubernetes.io/projected/e0540da8-e3fe-425f-9634-0791ba686651-kube-api-access-v482w\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.620922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.620914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.622318 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.621615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0540da8-e3fe-425f-9634-0791ba686651-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.622318 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.621759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0540da8-e3fe-425f-9634-0791ba686651-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.622318 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.622220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0540da8-e3fe-425f-9634-0791ba686651-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.623456 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.623410    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.623456 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.623426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0540da8-e3fe-425f-9634-0791ba686651-config-out\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.623613 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.623594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.623740 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.623723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0540da8-e3fe-425f-9634-0791ba686651-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.623908 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.623887    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.624052 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.624029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.624621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.624600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.624863 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.624847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.625165 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.625146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0540da8-e3fe-425f-9634-0791ba686651-web-config\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.629756 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.629736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v482w\" (UniqueName: \"kubernetes.io/projected/e0540da8-e3fe-425f-9634-0791ba686651-kube-api-access-v482w\") pod \"alertmanager-main-0\" (UID: \"e0540da8-e3fe-425f-9634-0791ba686651\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.774296 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.774249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 12:37:30.904991 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:30.904970    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 12:37:30.907402 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:37:30.907358    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0540da8_e3fe_425f_9634_0791ba686651.slice/crio-ae33e8ea4dab42a8859c00282bbf713f32f7f6340520a423cded6561329ac34e WatchSource:0}: Error finding container ae33e8ea4dab42a8859c00282bbf713f32f7f6340520a423cded6561329ac34e: Status 404 returned error can't find the container with id ae33e8ea4dab42a8859c00282bbf713f32f7f6340520a423cded6561329ac34e
Jul 28 12:37:31.397235 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:31.397200    2576 generic.go:356] "Generic (PLEG): container finished" podID="e0540da8-e3fe-425f-9634-0791ba686651" containerID="a93aa14ab0c6030fc28133d8e559cdbb914c5f2b30917b2275fb42e322e04cf4" exitCode=0
Jul 28 12:37:31.397390 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:31.397294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerDied","Data":"a93aa14ab0c6030fc28133d8e559cdbb914c5f2b30917b2275fb42e322e04cf4"}
Jul 28 12:37:31.397466 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:31.397408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"ae33e8ea4dab42a8859c00282bbf713f32f7f6340520a423cded6561329ac34e"}
Jul 28 12:37:31.721333 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:31.721310    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5784be4c-eebd-4ee5-83dd-fcf383439637" path="/var/lib/kubelet/pods/5784be4c-eebd-4ee5-83dd-fcf383439637/volumes"
Jul 28 12:37:32.405782 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.405749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"f296fafc2f31875acb535f946377a198cac7c86d4d94a774b7c65aa0b5c84022"}
Jul 28 12:37:32.406112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.405814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"30664d7cf5637206aeed96e78d6ce6493ca77573c75c61051e8575637f8954b6"}
Jul 28 12:37:32.406112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.405842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"0d0f984da61367c2cc47939d852c9b2c4a9f7ee8b352149fe0131ce59aa18f4d"}
Jul 28 12:37:32.406112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.405865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"2eb39f98e6728b678fea8e201f34543deca64b2aecdc85ca47db05d0746874fc"}
Jul 28 12:37:32.406112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.405895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"9a7ab41034f1447fb12a867f8e38bc067b946647b37301d30ea5b74ea9190952"}
Jul 28 12:37:32.406112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.405920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0540da8-e3fe-425f-9634-0791ba686651","Type":"ContainerStarted","Data":"bc1ef17013cb429f59d0055384329d9bee8f8a48585f4f7f0f7e2244f694f529"}
Jul 28 12:37:32.441143 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.441100    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.428216332 podStartE2EDuration="2.441083067s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.866735ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:37:32.439233392 +0000 UTC m=+145.340377810" watchObservedRunningTime="2026-07-28 12:37:32.441083067 +0000 UTC m=+145.342227455"
Jul 28 12:37:32.724878 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.724798    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:37:32.725150 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.725114    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="prometheus" containerID="cri-o://3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d" gracePeriod=600
Jul 28 12:37:32.725298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.725136    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy" containerID="cri-o://8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5" gracePeriod=600
Jul 28 12:37:32.725298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.725163    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-thanos" containerID="cri-o://96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e" gracePeriod=600
Jul 28 12:37:32.725298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.725185    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-web" containerID="cri-o://324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4" gracePeriod=600
Jul 28 12:37:32.725298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.725178    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="config-reloader" containerID="cri-o://25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642" gracePeriod=600
Jul 28 12:37:32.725298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.725163    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="thanos-sidecar" containerID="cri-o://91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa" gracePeriod=600
Jul 28 12:37:32.972419 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:32.972390    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.040877 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040809    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-trusted-ca-bundle\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.040877 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040850    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-db\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qrdq\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-kube-api-access-8qrdq\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040911    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-kubelet-serving-ca-bundle\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040940    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040973    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-tls\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.040997    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-metrics-client-certs\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041022    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-1\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-metrics-client-ca\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041075    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-config\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041131    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-tls-assets\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-grpc-tls\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041201    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-0\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041232    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-web-config\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041229    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041271    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-2\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-config-out\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041330    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-serving-certs-ca-bundle\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-thanos-prometheus-http-client-file\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-kube-rbac-proxy\") pod \"5c034273-5972-4ca0-a34d-41753e620f98\" (UID: \"5c034273-5972-4ca0-a34d-41753e620f98\") "
Jul 28 12:37:33.041874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041455    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.041874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041700    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.041874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041718    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.041874 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.041821    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.042109 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.042047    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.042165 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.042120    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.042211 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.042194    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.042745 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.042712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:37:33.043902 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.043861    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:37:33.045302 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.045249    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.045416 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.045363    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-kube-api-access-8qrdq" (OuterVolumeSpecName: "kube-api-access-8qrdq") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "kube-api-access-8qrdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:33.045416 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.045400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.045563 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.045426    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:37:33.045563 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.045545    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.046162 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.046125    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.046533 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.046506    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.046618 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.046567    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.046686 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.046626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.046741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.046715    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-config-out" (OuterVolumeSpecName: "config-out") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:37:33.046930 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.046914    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-config" (OuterVolumeSpecName: "config") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.055306 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.055283    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-web-config" (OuterVolumeSpecName: "web-config") pod "5c034273-5972-4ca0-a34d-41753e620f98" (UID: "5c034273-5972-4ca0-a34d-41753e620f98"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:37:33.142852 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142828    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-config-out\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142853    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142867    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142881    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-kube-rbac-proxy\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142894    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-db\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142906    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrdq\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-kube-api-access-8qrdq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142919    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.142936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142932    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142945    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-metrics-client-certs\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142958    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142971    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-configmap-metrics-client-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142984    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.142998    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.143011    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c034273-5972-4ca0-a34d-41753e620f98-tls-assets\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.143023    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-secret-grpc-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.143034    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.143046    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c034273-5972-4ca0-a34d-41753e620f98-web-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.143197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.143058    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c034273-5972-4ca0-a34d-41753e620f98-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:37:33.414248 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414215    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e" exitCode=0
Jul 28 12:37:33.414248 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414244    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5" exitCode=0
Jul 28 12:37:33.414248 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414252    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4" exitCode=0
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414260    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa" exitCode=0
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414268    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642" exitCode=0
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414275    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c034273-5972-4ca0-a34d-41753e620f98" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d" exitCode=0
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"}
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414322    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"}
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"}
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"}
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414494    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"}
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"}
Jul 28 12:37:33.414638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.414633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c034273-5972-4ca0-a34d-41753e620f98","Type":"ContainerDied","Data":"8bbe38b00cc8a917249b962c8c907ad32ba077bbd917bc85d1073aa34d67b6ab"}
Jul 28 12:37:33.423409 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.423392    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.432745 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.432724    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.439871 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.439852    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.447490 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.447468    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.448675 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.448497    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:37:33.455122 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.455099    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.456524 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.456504    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:37:33.473193 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.473171    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.479530 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.479486    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.479798 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.479778    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.479854 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.479805    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"} err="failed to get container status \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist"
Jul 28 12:37:33.479854 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.479821    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.480061 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.480045    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.480104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480068    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"} err="failed to get container status \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist"
Jul 28 12:37:33.480104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480083    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.480305 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.480289    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.480340 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480311    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"} err="failed to get container status \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist"
Jul 28 12:37:33.480340 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480328    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.480605 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.480584    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.480679 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480607    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"} err="failed to get container status \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist"
Jul 28 12:37:33.480679 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480623    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.480871 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.480854    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.480906 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480877    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"} err="failed to get container status \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist"
Jul 28 12:37:33.480906 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.480893    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.481118 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.481102    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.481171 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481122    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"} err="failed to get container status \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist"
Jul 28 12:37:33.481171 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481137    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.481347 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:37:33.481332    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.481417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481350    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"} err="failed to get container status \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist"
Jul 28 12:37:33.481417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481385    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.481637 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481620    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"} err="failed to get container status \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist"
Jul 28 12:37:33.481677 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481637    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.481855 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481838    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"} err="failed to get container status \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist"
Jul 28 12:37:33.481899 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.481857    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.482084 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482067    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"} err="failed to get container status \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist"
Jul 28 12:37:33.482084 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482083    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.482295 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482280    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"} err="failed to get container status \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist"
Jul 28 12:37:33.482339 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482295    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.482511 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482494    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"} err="failed to get container status \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist"
Jul 28 12:37:33.482560 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482512    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.482714 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482695    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"} err="failed to get container status \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist"
Jul 28 12:37:33.482757 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482714    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.482932 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482912    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"} err="failed to get container status \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist"
Jul 28 12:37:33.482973 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.482933    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.483138 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483115    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"} err="failed to get container status \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist"
Jul 28 12:37:33.483212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483139    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.483330 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483312    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"} err="failed to get container status \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist"
Jul 28 12:37:33.483393 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483331    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.483551 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483534    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"} err="failed to get container status \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist"
Jul 28 12:37:33.483621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483553    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.483802 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483785    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"} err="failed to get container status \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist"
Jul 28 12:37:33.483849 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.483803    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.484018 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484000    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"} err="failed to get container status \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist"
Jul 28 12:37:33.484076 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484019    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.484238 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484222    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"} err="failed to get container status \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist"
Jul 28 12:37:33.484277 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484238    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.484470 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484450    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"} err="failed to get container status \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist"
Jul 28 12:37:33.484532 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484472    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.484678 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484663    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"} err="failed to get container status \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist"
Jul 28 12:37:33.484741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484680    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.484889 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484874    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"} err="failed to get container status \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist"
Jul 28 12:37:33.484953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.484890    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.485104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485086    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"} err="failed to get container status \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist"
Jul 28 12:37:33.485143 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485105    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.485325 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485309    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"} err="failed to get container status \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist"
Jul 28 12:37:33.485395 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485326    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.485599 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485582    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"} err="failed to get container status \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist"
Jul 28 12:37:33.485643 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485601    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.485792 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485776    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"} err="failed to get container status \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist"
Jul 28 12:37:33.485850 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.485794    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.486027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486012    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"} err="failed to get container status \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist"
Jul 28 12:37:33.486086 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486026    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.486230 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486213    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"} err="failed to get container status \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist"
Jul 28 12:37:33.486299 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486232    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.486458 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486443    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"} err="failed to get container status \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist"
Jul 28 12:37:33.486502 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486458    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.486668 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486650    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"} err="failed to get container status \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist"
Jul 28 12:37:33.486727 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486669    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.486858 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486843    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"} err="failed to get container status \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist"
Jul 28 12:37:33.486936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.486860    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.487042 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487024    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"} err="failed to get container status \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist"
Jul 28 12:37:33.487098 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487043    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.487239 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487224    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"} err="failed to get container status \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist"
Jul 28 12:37:33.487280 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487240    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.487435 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487418    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"} err="failed to get container status \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist"
Jul 28 12:37:33.487495 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487438    2576 scope.go:117] "RemoveContainer" containerID="96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"
Jul 28 12:37:33.487639 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487624    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e"} err="failed to get container status \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": rpc error: code = NotFound desc = could not find container \"96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e\": container with ID starting with 96f4e4c6270c2c34793395f6c5cb5e370e5c23442aac3505c9464c7ad42d157e not found: ID does not exist"
Jul 28 12:37:33.487673 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487640    2576 scope.go:117] "RemoveContainer" containerID="8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"
Jul 28 12:37:33.487858 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487839    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5"} err="failed to get container status \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": rpc error: code = NotFound desc = could not find container \"8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5\": container with ID starting with 8bbb93b6f5cb49cc28a4cff21dd02fc7f814df9f5a1cd888952d212e01f632a5 not found: ID does not exist"
Jul 28 12:37:33.487924 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.487859    2576 scope.go:117] "RemoveContainer" containerID="324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"
Jul 28 12:37:33.488079 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488064    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4"} err="failed to get container status \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": rpc error: code = NotFound desc = could not find container \"324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4\": container with ID starting with 324b67f155a358f895de36199a83621b3f4d13c7e327ab1409ad86ed7b79c1e4 not found: ID does not exist"
Jul 28 12:37:33.488120 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488079    2576 scope.go:117] "RemoveContainer" containerID="91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"
Jul 28 12:37:33.488269 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488245    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa"} err="failed to get container status \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": rpc error: code = NotFound desc = could not find container \"91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa\": container with ID starting with 91c5783c95f17cad14921867d23b68e809669d3eb47d60e9e3dbf7ae207754fa not found: ID does not exist"
Jul 28 12:37:33.488269 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488266    2576 scope.go:117] "RemoveContainer" containerID="25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"
Jul 28 12:37:33.488490 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488473    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642"} err="failed to get container status \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": rpc error: code = NotFound desc = could not find container \"25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642\": container with ID starting with 25003315078aecb3d024d3181faad6df3e772d5bb0abd4c921587094fc906642 not found: ID does not exist"
Jul 28 12:37:33.488584 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488492    2576 scope.go:117] "RemoveContainer" containerID="3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"
Jul 28 12:37:33.488767 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488746    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d"} err="failed to get container status \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": rpc error: code = NotFound desc = could not find container \"3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d\": container with ID starting with 3a615c998a2bc0cdfdbe047a0140191f5428471aa1c931559cc4efac823e894d not found: ID does not exist"
Jul 28 12:37:33.488896 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.488769    2576 scope.go:117] "RemoveContainer" containerID="95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"
Jul 28 12:37:33.489264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.489235    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93"} err="failed to get container status \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": rpc error: code = NotFound desc = could not find container \"95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93\": container with ID starting with 95ddfcd8c619338442abfad371a662c9746096a13808edde58bc59f2e4f17d93 not found: ID does not exist"
Jul 28 12:37:33.490703 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490684    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:37:33.490951 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490940    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="prometheus"
Jul 28 12:37:33.490997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490952    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="prometheus"
Jul 28 12:37:33.490997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490962    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-web"
Jul 28 12:37:33.490997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490968    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-web"
Jul 28 12:37:33.490997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490981    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy"
Jul 28 12:37:33.490997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490987    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy"
Jul 28 12:37:33.490997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.490998    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="thanos-sidecar"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491003    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="thanos-sidecar"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491011    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-thanos"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491016    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-thanos"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491025    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="init-config-reloader"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491031    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="init-config-reloader"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491037    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="config-reloader"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491042    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="config-reloader"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491086    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="thanos-sidecar"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491094    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="prometheus"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491099    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-web"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491105    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="config-reloader"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491110    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy-thanos"
Jul 28 12:37:33.491221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.491128    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c034273-5972-4ca0-a34d-41753e620f98" containerName="kube-rbac-proxy"
Jul 28 12:37:33.496261 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.496245    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.502013 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.501980    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 12:37:33.502604 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.502585    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 12:37:33.502706 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.502586    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 12:37:33.502706 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.502682    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-34b9ptsam9mqu\""
Jul 28 12:37:33.502706 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.502694    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 12:37:33.503030 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 12:37:33.503118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503089    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 12:37:33.503175 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503094    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 12:37:33.503233 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503175    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 12:37:33.503233 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503088    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 12:37:33.503327 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503242    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-54m2v\""
Jul 28 12:37:33.503327 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503318    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 12:37:33.503454 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503443    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 12:37:33.503515 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.503459    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 12:37:33.505768 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.505752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 12:37:33.511637 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.511620    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 12:37:33.517072 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.517054    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:37:33.646419 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646518 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646557 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f05cd91a-2eb1-45ba-9840-701f401eb1d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646557 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-config\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646716 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646716 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646716 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646852 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646852 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646852 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f05cd91a-2eb1-45ba-9840-701f401eb1d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.646852 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.647034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.647034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.647034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98hc\" (UniqueName: \"kubernetes.io/projected/f05cd91a-2eb1-45ba-9840-701f401eb1d5-kube-api-access-h98hc\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.647034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.646975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.647034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.647003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.647034 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.647032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.719444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.719359    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c034273-5972-4ca0-a34d-41753e620f98" path="/var/lib/kubelet/pods/5c034273-5972-4ca0-a34d-41753e620f98/volumes"
Jul 28 12:37:33.747361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747430 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747430 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f05cd91a-2eb1-45ba-9840-701f401eb1d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747506 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747506 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747506 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747623 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-config\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747623 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747623 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f05cd91a-2eb1-45ba-9840-701f401eb1d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.747807 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.748118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.748118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98hc\" (UniqueName: \"kubernetes.io/projected/f05cd91a-2eb1-45ba-9840-701f401eb1d5-kube-api-access-h98hc\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.748118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.748118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.748118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.747920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.749197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.748404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.749197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.748434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.749197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.748968    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.749197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.749065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.749789 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.749760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.750565 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.750515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.751024 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.750995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f05cd91a-2eb1-45ba-9840-701f401eb1d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752410 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752410 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f05cd91a-2eb1-45ba-9840-701f401eb1d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752767 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752868 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.752925 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.752876    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-config\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.753445 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.753418    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f05cd91a-2eb1-45ba-9840-701f401eb1d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.753629 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.753603    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.753820 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.753798    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.753882 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.753834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.754234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.754215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f05cd91a-2eb1-45ba-9840-701f401eb1d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.759440 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.759419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98hc\" (UniqueName: \"kubernetes.io/projected/f05cd91a-2eb1-45ba-9840-701f401eb1d5-kube-api-access-h98hc\") pod \"prometheus-k8s-0\" (UID: \"f05cd91a-2eb1-45ba-9840-701f401eb1d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.805463 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.805442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:33.945861 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:33.945836    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 12:37:33.947668 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:37:33.947646    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05cd91a_2eb1_45ba_9840_701f401eb1d5.slice/crio-478b5cbc9f0fecc0a097eadd60a2e390c429657ea0d961a4e07d68447f23d2fd WatchSource:0}: Error finding container 478b5cbc9f0fecc0a097eadd60a2e390c429657ea0d961a4e07d68447f23d2fd: Status 404 returned error can't find the container with id 478b5cbc9f0fecc0a097eadd60a2e390c429657ea0d961a4e07d68447f23d2fd
Jul 28 12:37:34.420868 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:34.420832    2576 generic.go:356] "Generic (PLEG): container finished" podID="f05cd91a-2eb1-45ba-9840-701f401eb1d5" containerID="6564e8dadf0fbb1b9ef41f46b9726187952d28fffe752256958605414f8ec45c" exitCode=0
Jul 28 12:37:34.421219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:34.420924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerDied","Data":"6564e8dadf0fbb1b9ef41f46b9726187952d28fffe752256958605414f8ec45c"}
Jul 28 12:37:34.421219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:34.421043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"478b5cbc9f0fecc0a097eadd60a2e390c429657ea0d961a4e07d68447f23d2fd"}
Jul 28 12:37:35.430915 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.430876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"4b99dc6005ef08ce145217e171f46c01aedeef24732b47cb186724d953f9e886"}
Jul 28 12:37:35.431290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.430947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"b3a3d8121e38893ab66a4997c815b0570ed656dc39c0703ed2f49855cda08cfd"}
Jul 28 12:37:35.431290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.430974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"ee3533b6f43145433bfe13698a525d1912871fe66eacbd6eb90e3482327109c0"}
Jul 28 12:37:35.431290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.431001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"406145e057d18a28db7841e805a24355c39d93e658b27bfead0955d4487e9939"}
Jul 28 12:37:35.431290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.431032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"742c96614c11a87dbfbf39243007be8fe1d62aecd7db7f4959e2d8f99e3f4f2d"}
Jul 28 12:37:35.431290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.431057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f05cd91a-2eb1-45ba-9840-701f401eb1d5","Type":"ContainerStarted","Data":"ce5268043f204b4e5b19958b876756b9941be5da3669aa1cb80c99f3955970e5"}
Jul 28 12:37:35.463033 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:35.462980    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.447251224 podStartE2EDuration="2.46296065s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.709426ms" isStatefulPod=false podCreationTimestamp="2026-07-28 12:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:37:35.46046326 +0000 UTC m=+148.361607650" watchObservedRunningTime="2026-07-28 12:37:35.46296065 +0000 UTC m=+148.364105039"
Jul 28 12:37:38.806570 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:38.806539    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:37:49.073071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:37:49.073031    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95c7c8b48-2xnx4"]
Jul 28 12:38:14.092620 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.092554    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-95c7c8b48-2xnx4" podUID="b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" containerName="console" containerID="cri-o://7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107" gracePeriod=15
Jul 28 12:38:14.328296 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.328272    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:38:14.347704 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.347653    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-service-ca\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.347704 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.347683    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-config\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.347704 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.347702    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-trusted-ca-bundle\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.347967 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.347731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bmww\" (UniqueName: \"kubernetes.io/projected/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-kube-api-access-9bmww\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.347967 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.347753    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-oauth-serving-cert\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.348311 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348097    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-service-ca" (OuterVolumeSpecName: "service-ca") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:14.348311 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348117    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-config" (OuterVolumeSpecName: "console-config") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:14.348311 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348125    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-serving-cert\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.348311 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348177    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-oauth-config\") pod \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\" (UID: \"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad\") "
Jul 28 12:38:14.348311 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:14.348621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348458    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-service-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.348621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348476    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.348621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348485    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:38:14.348621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.348489    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.350518 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.350470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:14.351023 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.350986    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:38:14.351127 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.351084    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-kube-api-access-9bmww" (OuterVolumeSpecName: "kube-api-access-9bmww") pod "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" (UID: "b28e3dff-fffa-4a9e-bcc1-36efdccc0fad"). InnerVolumeSpecName "kube-api-access-9bmww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:38:14.448837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.448810    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.448837 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.448833    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-console-oauth-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.448958 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.448848    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bmww\" (UniqueName: \"kubernetes.io/projected/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-kube-api-access-9bmww\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.448958 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.448860    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad-oauth-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:38:14.596446 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.596414    2576 generic.go:356] "Generic (PLEG): container finished" podID="b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" containerID="7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107" exitCode=0
Jul 28 12:38:14.596567 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.596453    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7c8b48-2xnx4" event={"ID":"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad","Type":"ContainerDied","Data":"7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107"}
Jul 28 12:38:14.596567 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.596471    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95c7c8b48-2xnx4"
Jul 28 12:38:14.596567 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.596525    2576 scope.go:117] "RemoveContainer" containerID="7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107"
Jul 28 12:38:14.596712 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.596513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95c7c8b48-2xnx4" event={"ID":"b28e3dff-fffa-4a9e-bcc1-36efdccc0fad","Type":"ContainerDied","Data":"cc9fff7a5476db4a2f7e354e24dd07b9f74b3cfe4a29c3c3511d91ef83228022"}
Jul 28 12:38:14.605676 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.605648    2576 scope.go:117] "RemoveContainer" containerID="7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107"
Jul 28 12:38:14.605972 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:38:14.605954    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107\": container with ID starting with 7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107 not found: ID does not exist" containerID="7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107"
Jul 28 12:38:14.606017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.605981    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107"} err="failed to get container status \"7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107\": rpc error: code = NotFound desc = could not find container \"7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107\": container with ID starting with 7e20b28d8c24578a0b83612407592455490b254762c0fc6ee4806ec6687d2107 not found: ID does not exist"
Jul 28 12:38:14.618112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.618088    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95c7c8b48-2xnx4"]
Jul 28 12:38:14.621742 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:14.621720    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95c7c8b48-2xnx4"]
Jul 28 12:38:15.719261 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:15.719228    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" path="/var/lib/kubelet/pods/b28e3dff-fffa-4a9e-bcc1-36efdccc0fad/volumes"
Jul 28 12:38:33.806212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:33.806175    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:38:33.822340 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:33.822315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:38:34.692334 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:38:34.692311    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 12:40:07.513759 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:07.513720    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 12:40:16.349182 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.349140    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"]
Jul 28 12:40:16.351802 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.349476    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" containerName="console"
Jul 28 12:40:16.351802 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.349489    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" containerName="console"
Jul 28 12:40:16.351802 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.349555    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28e3dff-fffa-4a9e-bcc1-36efdccc0fad" containerName="console"
Jul 28 12:40:16.352648 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.352627    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.356409 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.356129    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v8fjp\""
Jul 28 12:40:16.364692 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.364668    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 12:40:16.364955 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.364882    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 12:40:16.367290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.367257    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"]
Jul 28 12:40:16.407314 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.407288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nr49\" (UniqueName: \"kubernetes.io/projected/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-kube-api-access-8nr49\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.407425 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.407319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.407425 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.407352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.508447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.508418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.508550 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.508487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nr49\" (UniqueName: \"kubernetes.io/projected/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-kube-api-access-8nr49\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.508550 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.508520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.508826 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.508801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.508862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.508826    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.517294 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.517264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nr49\" (UniqueName: \"kubernetes.io/projected/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-kube-api-access-8nr49\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.665851 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.665794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:16.795172 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.795144    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"]
Jul 28 12:40:16.797995 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:40:16.797971    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a4a10e_470f_405a_bc4c_f16b6df8e0b9.slice/crio-3e176b7bd040c338c32400bb739572113fabb9f6bbc3250cbfc9579ee50885cd WatchSource:0}: Error finding container 3e176b7bd040c338c32400bb739572113fabb9f6bbc3250cbfc9579ee50885cd: Status 404 returned error can't find the container with id 3e176b7bd040c338c32400bb739572113fabb9f6bbc3250cbfc9579ee50885cd
Jul 28 12:40:16.799902 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:16.799886    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:40:17.077404 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:17.077359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv" event={"ID":"60a4a10e-470f-405a-bc4c-f16b6df8e0b9","Type":"ContainerStarted","Data":"3e176b7bd040c338c32400bb739572113fabb9f6bbc3250cbfc9579ee50885cd"}
Jul 28 12:40:23.104486 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:23.104457    2576 generic.go:356] "Generic (PLEG): container finished" podID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerID="8fad7ee362e59fe4f4db01cc28b8b46b70cfa4503a032b0df88771d9d9cfedce" exitCode=0
Jul 28 12:40:23.104813 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:23.104551    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv" event={"ID":"60a4a10e-470f-405a-bc4c-f16b6df8e0b9","Type":"ContainerDied","Data":"8fad7ee362e59fe4f4db01cc28b8b46b70cfa4503a032b0df88771d9d9cfedce"}
Jul 28 12:40:26.118822 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:26.118768    2576 generic.go:356] "Generic (PLEG): container finished" podID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerID="e925a70bcb80308e04b79f0f490d956683e4d450c9f7b7c6bf8fb4d082e6b341" exitCode=0
Jul 28 12:40:26.119220 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:26.118850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv" event={"ID":"60a4a10e-470f-405a-bc4c-f16b6df8e0b9","Type":"ContainerDied","Data":"e925a70bcb80308e04b79f0f490d956683e4d450c9f7b7c6bf8fb4d082e6b341"}
Jul 28 12:40:33.149359 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:33.149332    2576 generic.go:356] "Generic (PLEG): container finished" podID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerID="11413116358bf80ecbb886a48921278ebeca0fcf39b5eccb2823da0e3ab68afe" exitCode=0
Jul 28 12:40:33.149704 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:33.149408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv" event={"ID":"60a4a10e-470f-405a-bc4c-f16b6df8e0b9","Type":"ContainerDied","Data":"11413116358bf80ecbb886a48921278ebeca0fcf39b5eccb2823da0e3ab68afe"}
Jul 28 12:40:34.282831 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.282807    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:34.344657 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.344634    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-util\") pod \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") "
Jul 28 12:40:34.344779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.344701    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-bundle\") pod \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") "
Jul 28 12:40:34.344779 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.344740    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nr49\" (UniqueName: \"kubernetes.io/projected/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-kube-api-access-8nr49\") pod \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\" (UID: \"60a4a10e-470f-405a-bc4c-f16b6df8e0b9\") "
Jul 28 12:40:34.345223 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.345194    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-bundle" (OuterVolumeSpecName: "bundle") pod "60a4a10e-470f-405a-bc4c-f16b6df8e0b9" (UID: "60a4a10e-470f-405a-bc4c-f16b6df8e0b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:40:34.346699 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.346671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-kube-api-access-8nr49" (OuterVolumeSpecName: "kube-api-access-8nr49") pod "60a4a10e-470f-405a-bc4c-f16b6df8e0b9" (UID: "60a4a10e-470f-405a-bc4c-f16b6df8e0b9"). InnerVolumeSpecName "kube-api-access-8nr49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:40:34.348737 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.348716    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-util" (OuterVolumeSpecName: "util") pod "60a4a10e-470f-405a-bc4c-f16b6df8e0b9" (UID: "60a4a10e-470f-405a-bc4c-f16b6df8e0b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:40:34.446104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.446059    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nr49\" (UniqueName: \"kubernetes.io/projected/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-kube-api-access-8nr49\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:40:34.446104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.446077    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-util\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:40:34.446104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:34.446086    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60a4a10e-470f-405a-bc4c-f16b6df8e0b9-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:40:35.159443 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:35.159405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv" event={"ID":"60a4a10e-470f-405a-bc4c-f16b6df8e0b9","Type":"ContainerDied","Data":"3e176b7bd040c338c32400bb739572113fabb9f6bbc3250cbfc9579ee50885cd"}
Jul 28 12:40:35.159619 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:35.159488    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e176b7bd040c338c32400bb739572113fabb9f6bbc3250cbfc9579ee50885cd"
Jul 28 12:40:35.159619 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:35.159422    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b57dstv"
Jul 28 12:40:43.828138 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828107    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"]
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828399    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="extract"
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828410    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="extract"
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828419    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="util"
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828424    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="util"
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828440    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="pull"
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828445    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="pull"
Jul 28 12:40:43.828487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.828491    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a4a10e-470f-405a-bc4c-f16b6df8e0b9" containerName="extract"
Jul 28 12:40:43.831815 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.831800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:43.834637 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.834617    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 12:40:43.835870 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.835849    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 12:40:43.835982 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.835876    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-j8mcg\""
Jul 28 12:40:43.835982 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.835876    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 12:40:43.835982 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.835855    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 12:40:43.835982 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.835976    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 12:40:43.840127 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.840108    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"]
Jul 28 12:40:43.916160 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.916137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e34148d3-adb9-46cb-8978-2b7d876761cd-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:43.916256 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.916186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-certificates\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:43.916256 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:43.916248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trs92\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-kube-api-access-trs92\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.017268 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.017243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-certificates\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.017390 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.017284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trs92\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-kube-api-access-trs92\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.017390 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.017334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e34148d3-adb9-46cb-8978-2b7d876761cd-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.017471 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:40:44.017416    2576 secret.go:281] references non-existent secret key: tls.crt
Jul 28 12:40:44.017471 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:40:44.017436    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 12:40:44.017471 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:40:44.017457    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp: references non-existent secret key: tls.crt
Jul 28 12:40:44.017562 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:40:44.017531    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-certificates podName:e34148d3-adb9-46cb-8978-2b7d876761cd nodeName:}" failed. No retries permitted until 2026-07-28 12:40:44.517511933 +0000 UTC m=+337.418656301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-certificates") pod "keda-metrics-apiserver-55557df8c-zvzxp" (UID: "e34148d3-adb9-46cb-8978-2b7d876761cd") : references non-existent secret key: tls.crt
Jul 28 12:40:44.017760 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.017739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e34148d3-adb9-46cb-8978-2b7d876761cd-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.026663 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.026645    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trs92\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-kube-api-access-trs92\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.431800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.431768    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-b6k79"]
Jul 28 12:40:44.435605 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.435587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.438083 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.438059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 12:40:44.449185 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.449158    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-b6k79"]
Jul 28 12:40:44.520925 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.520897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9wd\" (UniqueName: \"kubernetes.io/projected/0210129c-5011-40cf-a5ac-d73ec749a770-kube-api-access-pj9wd\") pod \"keda-admission-56f9798bb4-b6k79\" (UID: \"0210129c-5011-40cf-a5ac-d73ec749a770\") " pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.521030 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.520950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-certificates\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.521030 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.521009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0210129c-5011-40cf-a5ac-d73ec749a770-certificates\") pod \"keda-admission-56f9798bb4-b6k79\" (UID: \"0210129c-5011-40cf-a5ac-d73ec749a770\") " pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.523445 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.523426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e34148d3-adb9-46cb-8978-2b7d876761cd-certificates\") pod \"keda-metrics-apiserver-55557df8c-zvzxp\" (UID: \"e34148d3-adb9-46cb-8978-2b7d876761cd\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.622272 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.622244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9wd\" (UniqueName: \"kubernetes.io/projected/0210129c-5011-40cf-a5ac-d73ec749a770-kube-api-access-pj9wd\") pod \"keda-admission-56f9798bb4-b6k79\" (UID: \"0210129c-5011-40cf-a5ac-d73ec749a770\") " pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.622366 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.622297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0210129c-5011-40cf-a5ac-d73ec749a770-certificates\") pod \"keda-admission-56f9798bb4-b6k79\" (UID: \"0210129c-5011-40cf-a5ac-d73ec749a770\") " pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.624561 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.624536    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0210129c-5011-40cf-a5ac-d73ec749a770-certificates\") pod \"keda-admission-56f9798bb4-b6k79\" (UID: \"0210129c-5011-40cf-a5ac-d73ec749a770\") " pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.630101 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.630075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9wd\" (UniqueName: \"kubernetes.io/projected/0210129c-5011-40cf-a5ac-d73ec749a770-kube-api-access-pj9wd\") pod \"keda-admission-56f9798bb4-b6k79\" (UID: \"0210129c-5011-40cf-a5ac-d73ec749a770\") " pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.742798 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.742740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:44.745848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.745825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:44.878886 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.878776    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"]
Jul 28 12:40:44.881634 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:40:44.881601    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34148d3_adb9_46cb_8978_2b7d876761cd.slice/crio-1558c3a34b30052c3be0749160065e3a28e873b518b2f06f3775441092e4bfaf WatchSource:0}: Error finding container 1558c3a34b30052c3be0749160065e3a28e873b518b2f06f3775441092e4bfaf: Status 404 returned error can't find the container with id 1558c3a34b30052c3be0749160065e3a28e873b518b2f06f3775441092e4bfaf
Jul 28 12:40:44.899444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:44.899425    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-b6k79"]
Jul 28 12:40:44.901207 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:40:44.901186    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0210129c_5011_40cf_a5ac_d73ec749a770.slice/crio-cb4dbcf2005c79e8014a27e2b115bdf88e0dbb9c418fc35221a4dd8f040d0e2b WatchSource:0}: Error finding container cb4dbcf2005c79e8014a27e2b115bdf88e0dbb9c418fc35221a4dd8f040d0e2b: Status 404 returned error can't find the container with id cb4dbcf2005c79e8014a27e2b115bdf88e0dbb9c418fc35221a4dd8f040d0e2b
Jul 28 12:40:45.206066 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:45.206034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp" event={"ID":"e34148d3-adb9-46cb-8978-2b7d876761cd","Type":"ContainerStarted","Data":"1558c3a34b30052c3be0749160065e3a28e873b518b2f06f3775441092e4bfaf"}
Jul 28 12:40:45.207069 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:45.207046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-b6k79" event={"ID":"0210129c-5011-40cf-a5ac-d73ec749a770","Type":"ContainerStarted","Data":"cb4dbcf2005c79e8014a27e2b115bdf88e0dbb9c418fc35221a4dd8f040d0e2b"}
Jul 28 12:40:48.222442 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:48.222406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-b6k79" event={"ID":"0210129c-5011-40cf-a5ac-d73ec749a770","Type":"ContainerStarted","Data":"3a4cf2d13062f5ec1882df41cdd934d5dbab7b70fceaf05129f8b63b39665b14"}
Jul 28 12:40:48.222854 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:48.222468    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:40:48.223864 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:48.223841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp" event={"ID":"e34148d3-adb9-46cb-8978-2b7d876761cd","Type":"ContainerStarted","Data":"1bdc58d6f4f6c7e6c730f495cf04406938906a6cf0859ae88781e80f4bddf4b1"}
Jul 28 12:40:48.223983 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:48.223898    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:40:48.239563 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:48.239518    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-b6k79" podStartSLOduration=1.66074963 podStartE2EDuration="4.239506721s" totalImagesPullingTime="2.578757091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:48.239172659 +0000 UTC m=+341.140317048" watchObservedRunningTime="2026-07-28 12:40:48.239506721 +0000 UTC m=+341.140651145"
Jul 28 12:40:48.257046 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:48.257011    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp" podStartSLOduration=2.651927768 podStartE2EDuration="5.256999293s" totalImagesPullingTime="2.605071525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:40:48.255153204 +0000 UTC m=+341.156297592" watchObservedRunningTime="2026-07-28 12:40:48.256999293 +0000 UTC m=+341.158143680"
Jul 28 12:40:59.232364 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:40:59.232274    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-zvzxp"
Jul 28 12:41:09.230707 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:41:09.230673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-b6k79"
Jul 28 12:42:02.252680 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.252643    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wgxln"]
Jul 28 12:42:02.256233 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.256211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.259226 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.259205    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-dr9qk\""
Jul 28 12:42:02.259389 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.259361    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 12:42:02.260572 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.260557    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 12:42:02.261213 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.261197    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 12:42:02.272844 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.272826    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wgxln"]
Jul 28 12:42:02.361197 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.361169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-cert\") pod \"kserve-controller-manager-68bc5447d6-wgxln\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") " pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.361289 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.361210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtds\" (UniqueName: \"kubernetes.io/projected/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-kube-api-access-vxtds\") pod \"kserve-controller-manager-68bc5447d6-wgxln\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") " pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.462116 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.462086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-cert\") pod \"kserve-controller-manager-68bc5447d6-wgxln\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") " pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.462235 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.462136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtds\" (UniqueName: \"kubernetes.io/projected/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-kube-api-access-vxtds\") pod \"kserve-controller-manager-68bc5447d6-wgxln\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") " pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.464487 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.464467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-cert\") pod \"kserve-controller-manager-68bc5447d6-wgxln\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") " pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.473300 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.473278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtds\" (UniqueName: \"kubernetes.io/projected/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-kube-api-access-vxtds\") pod \"kserve-controller-manager-68bc5447d6-wgxln\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") " pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.566419 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.566342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:02.691778 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:02.691753    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wgxln"]
Jul 28 12:42:02.693850 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:42:02.693828    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae2bb81_95e7_4ee9_8eb0_ae624d1ec8e2.slice/crio-bb9f01d274269cc47ba26e1f5e673f920c4d8f8fb3f25c750d25a9e64d2b4cfe WatchSource:0}: Error finding container bb9f01d274269cc47ba26e1f5e673f920c4d8f8fb3f25c750d25a9e64d2b4cfe: Status 404 returned error can't find the container with id bb9f01d274269cc47ba26e1f5e673f920c4d8f8fb3f25c750d25a9e64d2b4cfe
Jul 28 12:42:03.533135 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:03.533086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln" event={"ID":"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2","Type":"ContainerStarted","Data":"bb9f01d274269cc47ba26e1f5e673f920c4d8f8fb3f25c750d25a9e64d2b4cfe"}
Jul 28 12:42:05.542924 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:05.542884    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln" event={"ID":"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2","Type":"ContainerStarted","Data":"7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347"}
Jul 28 12:42:05.543451 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:05.542982    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:05.560348 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:05.560294    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln" podStartSLOduration=1.227417162 podStartE2EDuration="3.560278938s" totalImagesPullingTime="2.332861776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:05.559774789 +0000 UTC m=+418.460919177" watchObservedRunningTime="2026-07-28 12:42:05.560278938 +0000 UTC m=+418.461423378"
Jul 28 12:42:36.552723 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:36.552640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:42.790628 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.790595    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54548f458f-9z85b"]
Jul 28 12:42:42.795228 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.795207    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.799723 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.799698    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:42:42.801184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801140    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:42:42.801184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:42:42.801364 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:42:42.801364 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801274    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:42:42.801364 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801319    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:42:42.801554 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801153    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:42:42.801678 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.801662    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-924zr\""
Jul 28 12:42:42.806454 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.806436    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:42:42.810630 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.810606    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54548f458f-9z85b"]
Jul 28 12:42:42.852770 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-oauth-config\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.852873 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-serving-cert\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.852873 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-config\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.852873 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-oauth-serving-cert\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.852997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-service-ca\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.852997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t429\" (UniqueName: \"kubernetes.io/projected/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-kube-api-access-2t429\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.852997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.852987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-trusted-ca-bundle\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954137 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-oauth-config\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954272 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-serving-cert\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954272 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-config\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-oauth-serving-cert\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-service-ca\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t429\" (UniqueName: \"kubernetes.io/projected/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-kube-api-access-2t429\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.954417 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.954367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-trusted-ca-bundle\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.955112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.955085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-oauth-serving-cert\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.955221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.955085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-config\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.955221 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.955185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-service-ca\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.955423 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.955364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-trusted-ca-bundle\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.956882 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.956857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-serving-cert\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.956986 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.956899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-console-oauth-config\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:42.963822 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:42.963800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t429\" (UniqueName: \"kubernetes.io/projected/c49c9874-7c57-4cb2-babc-2ef08dca2bcd-kube-api-access-2t429\") pod \"console-54548f458f-9z85b\" (UID: \"c49c9874-7c57-4cb2-babc-2ef08dca2bcd\") " pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:43.104824 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:43.104793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:43.291636 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:43.291609    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54548f458f-9z85b"]
Jul 28 12:42:43.293727 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:42:43.293706    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49c9874_7c57_4cb2_babc_2ef08dca2bcd.slice/crio-4c0cd2d3d316c606bcfc9dd57b94f5c1a2a980236f65e9ae4dcd18711d1cef44 WatchSource:0}: Error finding container 4c0cd2d3d316c606bcfc9dd57b94f5c1a2a980236f65e9ae4dcd18711d1cef44: Status 404 returned error can't find the container with id 4c0cd2d3d316c606bcfc9dd57b94f5c1a2a980236f65e9ae4dcd18711d1cef44
Jul 28 12:42:43.697065 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:43.697035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54548f458f-9z85b" event={"ID":"c49c9874-7c57-4cb2-babc-2ef08dca2bcd","Type":"ContainerStarted","Data":"a8d9944d64318bac86a421045222494427a6a289780d6881cfbc0a7575c33e8b"}
Jul 28 12:42:43.697263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:43.697090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54548f458f-9z85b" event={"ID":"c49c9874-7c57-4cb2-babc-2ef08dca2bcd","Type":"ContainerStarted","Data":"4c0cd2d3d316c606bcfc9dd57b94f5c1a2a980236f65e9ae4dcd18711d1cef44"}
Jul 28 12:42:43.719238 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:43.719190    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54548f458f-9z85b" podStartSLOduration=1.719172519 podStartE2EDuration="1.719172519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:43.718422353 +0000 UTC m=+456.619566743" watchObservedRunningTime="2026-07-28 12:42:43.719172519 +0000 UTC m=+456.620316908"
Jul 28 12:42:46.275452 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.275402    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wgxln"]
Jul 28 12:42:46.275980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.275664    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln" podUID="7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" containerName="manager" containerID="cri-o://7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347" gracePeriod=10
Jul 28 12:42:46.297482 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.297454    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-7s6pb"]
Jul 28 12:42:46.300344 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.300322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.309437 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.309405    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-7s6pb"]
Jul 28 12:42:46.380591 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.380569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b068c65c-e076-4dfd-9ae4-778197a15297-cert\") pod \"kserve-controller-manager-68bc5447d6-7s6pb\" (UID: \"b068c65c-e076-4dfd-9ae4-778197a15297\") " pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.380700 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.380658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k642d\" (UniqueName: \"kubernetes.io/projected/b068c65c-e076-4dfd-9ae4-778197a15297-kube-api-access-k642d\") pod \"kserve-controller-manager-68bc5447d6-7s6pb\" (UID: \"b068c65c-e076-4dfd-9ae4-778197a15297\") " pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.481638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.481608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k642d\" (UniqueName: \"kubernetes.io/projected/b068c65c-e076-4dfd-9ae4-778197a15297-kube-api-access-k642d\") pod \"kserve-controller-manager-68bc5447d6-7s6pb\" (UID: \"b068c65c-e076-4dfd-9ae4-778197a15297\") " pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.481759 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.481676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b068c65c-e076-4dfd-9ae4-778197a15297-cert\") pod \"kserve-controller-manager-68bc5447d6-7s6pb\" (UID: \"b068c65c-e076-4dfd-9ae4-778197a15297\") " pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.483738 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.483714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b068c65c-e076-4dfd-9ae4-778197a15297-cert\") pod \"kserve-controller-manager-68bc5447d6-7s6pb\" (UID: \"b068c65c-e076-4dfd-9ae4-778197a15297\") " pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.490219 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.490201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k642d\" (UniqueName: \"kubernetes.io/projected/b068c65c-e076-4dfd-9ae4-778197a15297-kube-api-access-k642d\") pod \"kserve-controller-manager-68bc5447d6-7s6pb\" (UID: \"b068c65c-e076-4dfd-9ae4-778197a15297\") " pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.514729 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.514712    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:46.582817 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.582798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-cert\") pod \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") "
Jul 28 12:42:46.582925 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.582914    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxtds\" (UniqueName: \"kubernetes.io/projected/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-kube-api-access-vxtds\") pod \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\" (UID: \"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2\") "
Jul 28 12:42:46.584787 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.584759    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-kube-api-access-vxtds" (OuterVolumeSpecName: "kube-api-access-vxtds") pod "7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" (UID: "7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2"). InnerVolumeSpecName "kube-api-access-vxtds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:42:46.584787 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.584783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-cert" (OuterVolumeSpecName: "cert") pod "7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" (UID: "7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:42:46.650118 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.650085    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:46.683731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.683711    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxtds\" (UniqueName: \"kubernetes.io/projected/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-kube-api-access-vxtds\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:42:46.683731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.683732    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:42:46.712708 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.712683    2576 generic.go:356] "Generic (PLEG): container finished" podID="7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" containerID="7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347" exitCode=0
Jul 28 12:42:46.712823 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.712754    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln"
Jul 28 12:42:46.712823 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.712765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln" event={"ID":"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2","Type":"ContainerDied","Data":"7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347"}
Jul 28 12:42:46.712823 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.712821    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-wgxln" event={"ID":"7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2","Type":"ContainerDied","Data":"bb9f01d274269cc47ba26e1f5e673f920c4d8f8fb3f25c750d25a9e64d2b4cfe"}
Jul 28 12:42:46.712933 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.712844    2576 scope.go:117] "RemoveContainer" containerID="7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347"
Jul 28 12:42:46.724470 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.724452    2576 scope.go:117] "RemoveContainer" containerID="7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347"
Jul 28 12:42:46.724745 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:42:46.724721    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347\": container with ID starting with 7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347 not found: ID does not exist" containerID="7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347"
Jul 28 12:42:46.724831 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.724748    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347"} err="failed to get container status \"7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347\": rpc error: code = NotFound desc = could not find container \"7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347\": container with ID starting with 7cf2449dc2e50de5f753ca1152f574930e17f10fc5cf0a7d5757098f79e09347 not found: ID does not exist"
Jul 28 12:42:46.736227 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.736206    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wgxln"]
Jul 28 12:42:46.741304 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.741284    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-wgxln"]
Jul 28 12:42:46.778418 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:46.778403    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-68bc5447d6-7s6pb"]
Jul 28 12:42:46.780399 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:42:46.780359    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb068c65c_e076_4dfd_9ae4_778197a15297.slice/crio-bd2fa7d956233c548a34953c9e079562e0da98b1c9bdb470235e14c033c66e09 WatchSource:0}: Error finding container bd2fa7d956233c548a34953c9e079562e0da98b1c9bdb470235e14c033c66e09: Status 404 returned error can't find the container with id bd2fa7d956233c548a34953c9e079562e0da98b1c9bdb470235e14c033c66e09
Jul 28 12:42:47.720606 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:47.720568    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" path="/var/lib/kubelet/pods/7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2/volumes"
Jul 28 12:42:47.721071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:47.720877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:42:47.721071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:47.720928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb" event={"ID":"b068c65c-e076-4dfd-9ae4-778197a15297","Type":"ContainerStarted","Data":"ca991735f1c0b69b6f5a1234d3e17fe9382c3a5fe092af1eb7b94bd60840284d"}
Jul 28 12:42:47.721071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:47.720955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb" event={"ID":"b068c65c-e076-4dfd-9ae4-778197a15297","Type":"ContainerStarted","Data":"bd2fa7d956233c548a34953c9e079562e0da98b1c9bdb470235e14c033c66e09"}
Jul 28 12:42:47.736613 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:47.736565    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb" podStartSLOduration=1.268815169 podStartE2EDuration="1.736553008s" totalImagesPullingTime="467.737839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:42:47.73504205 +0000 UTC m=+460.636186454" watchObservedRunningTime="2026-07-28 12:42:47.736553008 +0000 UTC m=+460.637697396"
Jul 28 12:42:53.105265 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:53.105225    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:53.105706 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:53.105429    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:53.111168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:53.111134    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:42:53.744559 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:42:53.744525    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54548f458f-9z85b"
Jul 28 12:43:18.728402 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:18.728343    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-68bc5447d6-7s6pb"
Jul 28 12:43:38.064911 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.064871    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-g846t"]
Jul 28 12:43:38.065343 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.065212    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" containerName="manager"
Jul 28 12:43:38.065343 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.065223    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" containerName="manager"
Jul 28 12:43:38.065343 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.065286    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae2bb81-95e7-4ee9-8eb0-ae624d1ec8e2" containerName="manager"
Jul 28 12:43:38.068318 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.068301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g846t"
Jul 28 12:43:38.072155 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.072134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 12:43:38.072522 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.072497    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v9v8l\""
Jul 28 12:43:38.079615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.079590    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-g846t"]
Jul 28 12:43:38.162200 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.162173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-model-cache\") pod \"s3-init-g846t\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") " pod="kserve/s3-init-g846t"
Jul 28 12:43:38.162304 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.162212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl9j\" (UniqueName: \"kubernetes.io/projected/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-kube-api-access-7kl9j\") pod \"s3-init-g846t\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") " pod="kserve/s3-init-g846t"
Jul 28 12:43:38.262845 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.262816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-model-cache\") pod \"s3-init-g846t\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") " pod="kserve/s3-init-g846t"
Jul 28 12:43:38.262944 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.262854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl9j\" (UniqueName: \"kubernetes.io/projected/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-kube-api-access-7kl9j\") pod \"s3-init-g846t\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") " pod="kserve/s3-init-g846t"
Jul 28 12:43:38.263203 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.263184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-model-cache\") pod \"s3-init-g846t\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") " pod="kserve/s3-init-g846t"
Jul 28 12:43:38.272363 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.272333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl9j\" (UniqueName: \"kubernetes.io/projected/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-kube-api-access-7kl9j\") pod \"s3-init-g846t\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") " pod="kserve/s3-init-g846t"
Jul 28 12:43:38.391930 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.391902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g846t"
Jul 28 12:43:38.522400 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.522360    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-g846t"]
Jul 28 12:43:38.524384 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:43:38.524349    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb6ec68_5d6e_4cd3_a231_6399bd271ab7.slice/crio-ba3e4241ac50fcb071980884a8f4f4e9638ccb0de891550ccab878489d891c7d WatchSource:0}: Error finding container ba3e4241ac50fcb071980884a8f4f4e9638ccb0de891550ccab878489d891c7d: Status 404 returned error can't find the container with id ba3e4241ac50fcb071980884a8f4f4e9638ccb0de891550ccab878489d891c7d
Jul 28 12:43:38.918616 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:38.918583    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g846t" event={"ID":"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7","Type":"ContainerStarted","Data":"ba3e4241ac50fcb071980884a8f4f4e9638ccb0de891550ccab878489d891c7d"}
Jul 28 12:43:42.938420 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:42.938382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g846t" event={"ID":"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7","Type":"ContainerStarted","Data":"1480a7708a25c2aeef8b672ca0326224f2623ed0c555c65265a7af819fd689c4"}
Jul 28 12:43:43.943804 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:43.943768    2576 generic.go:356] "Generic (PLEG): container finished" podID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerID="1480a7708a25c2aeef8b672ca0326224f2623ed0c555c65265a7af819fd689c4" exitCode=0
Jul 28 12:43:43.944178 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:43.943858    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g846t" event={"ID":"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7","Type":"ContainerDied","Data":"1480a7708a25c2aeef8b672ca0326224f2623ed0c555c65265a7af819fd689c4"}
Jul 28 12:43:48.971473 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:48.971441    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g846t" event={"ID":"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7","Type":"ContainerStarted","Data":"f5651843f5435682bd1cca16b30217f8efadcf9f8475d33711b8b08ea274ca22"}
Jul 28 12:43:48.987775 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:43:48.987715    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-g846t" podStartSLOduration=1.510079587 podStartE2EDuration="10.987697909s" totalImagesPullingTime="7.811027374s" totalInitContainerRuntime="1.666590948s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:43:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:43:48.987266384 +0000 UTC m=+521.888410772" watchObservedRunningTime="2026-07-28 12:43:48.987697909 +0000 UTC m=+521.888842300"
Jul 28 12:44:53.240581 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:53.240518    2576 generic.go:356] "Generic (PLEG): container finished" podID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerID="f5651843f5435682bd1cca16b30217f8efadcf9f8475d33711b8b08ea274ca22" exitCode=0
Jul 28 12:44:53.240909 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:53.240588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g846t" event={"ID":"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7","Type":"ContainerDied","Data":"f5651843f5435682bd1cca16b30217f8efadcf9f8475d33711b8b08ea274ca22"}
Jul 28 12:44:54.368811 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.368787    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g846t"
Jul 28 12:44:54.445451 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.445427    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-model-cache\") pod \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") "
Jul 28 12:44:54.445571 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.445503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kl9j\" (UniqueName: \"kubernetes.io/projected/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-kube-api-access-7kl9j\") pod \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\" (UID: \"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7\") "
Jul 28 12:44:54.447332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.447305    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-kube-api-access-7kl9j" (OuterVolumeSpecName: "kube-api-access-7kl9j") pod "2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" (UID: "2cb6ec68-5d6e-4cd3-a231-6399bd271ab7"). InnerVolumeSpecName "kube-api-access-7kl9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:44:54.463887 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.463864    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" (UID: "2cb6ec68-5d6e-4cd3-a231-6399bd271ab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:44:54.546505 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.546458    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-model-cache\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:44:54.546505 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:54.546477    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kl9j\" (UniqueName: \"kubernetes.io/projected/2cb6ec68-5d6e-4cd3-a231-6399bd271ab7-kube-api-access-7kl9j\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:44:55.250092 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:55.250055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g846t" event={"ID":"2cb6ec68-5d6e-4cd3-a231-6399bd271ab7","Type":"ContainerDied","Data":"ba3e4241ac50fcb071980884a8f4f4e9638ccb0de891550ccab878489d891c7d"}
Jul 28 12:44:55.250092 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:55.250076    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g846t"
Jul 28 12:44:55.250092 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:44:55.250097    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba3e4241ac50fcb071980884a8f4f4e9638ccb0de891550ccab878489d891c7d"
Jul 28 12:45:03.304765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.304726    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-m7mhm"]
Jul 28 12:45:03.305276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.305092    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerName="download-hf-model"
Jul 28 12:45:03.305276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.305104    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerName="download-hf-model"
Jul 28 12:45:03.305276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.305120    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerName="s3-init"
Jul 28 12:45:03.305276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.305125    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerName="s3-init"
Jul 28 12:45:03.305276 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.305183    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb6ec68-5d6e-4cd3-a231-6399bd271ab7" containerName="s3-init"
Jul 28 12:45:03.308379 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.308349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.311836 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.311808    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v9v8l\""
Jul 28 12:45:03.313103 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.313065    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 12:45:03.318538 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.318506    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-m7mhm"]
Jul 28 12:45:03.416281 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.416249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-model-cache\") pod \"s3-tls-init-custom-m7mhm\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") " pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.416406 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.416307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqn7n\" (UniqueName: \"kubernetes.io/projected/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-kube-api-access-sqn7n\") pod \"s3-tls-init-custom-m7mhm\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") " pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.516864 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.516836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqn7n\" (UniqueName: \"kubernetes.io/projected/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-kube-api-access-sqn7n\") pod \"s3-tls-init-custom-m7mhm\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") " pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.516971 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.516893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-model-cache\") pod \"s3-tls-init-custom-m7mhm\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") " pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.517234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.517215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-model-cache\") pod \"s3-tls-init-custom-m7mhm\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") " pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.526123 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.526092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqn7n\" (UniqueName: \"kubernetes.io/projected/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-kube-api-access-sqn7n\") pod \"s3-tls-init-custom-m7mhm\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") " pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.633977 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.633952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:03.766262 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:03.766239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-m7mhm"]
Jul 28 12:45:03.767634 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:45:03.767613    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c70dd70_b158_4f4f_a049_c0ba22f5e2f9.slice/crio-f266959d75ab2d48b9c216eb9becace611d37501feded2dc3a68f8392e386117 WatchSource:0}: Error finding container f266959d75ab2d48b9c216eb9becace611d37501feded2dc3a68f8392e386117: Status 404 returned error can't find the container with id f266959d75ab2d48b9c216eb9becace611d37501feded2dc3a68f8392e386117
Jul 28 12:45:04.300336 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:04.300303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-m7mhm" event={"ID":"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9","Type":"ContainerStarted","Data":"39df46321c8cff88cb462e9c14563dcc1dfd510a9cbef345008083df5f6c8e5e"}
Jul 28 12:45:04.300527 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:04.300347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-m7mhm" event={"ID":"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9","Type":"ContainerStarted","Data":"f266959d75ab2d48b9c216eb9becace611d37501feded2dc3a68f8392e386117"}
Jul 28 12:45:06.311493 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:06.311457    2576 generic.go:356] "Generic (PLEG): container finished" podID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerID="39df46321c8cff88cb462e9c14563dcc1dfd510a9cbef345008083df5f6c8e5e" exitCode=0
Jul 28 12:45:06.311886 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:06.311535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-m7mhm" event={"ID":"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9","Type":"ContainerDied","Data":"39df46321c8cff88cb462e9c14563dcc1dfd510a9cbef345008083df5f6c8e5e"}
Jul 28 12:45:07.318477 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:07.318442    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-m7mhm" event={"ID":"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9","Type":"ContainerStarted","Data":"58b5a94fa7b75407bcf1b820e5305b3f5ef3c6855144efa78840a037738593a5"}
Jul 28 12:45:07.336457 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:07.336412    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-m7mhm" podStartSLOduration=2.766069411 podStartE2EDuration="4.336396071s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.57032666s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:45:07.336331048 +0000 UTC m=+600.237475437" watchObservedRunningTime="2026-07-28 12:45:07.336396071 +0000 UTC m=+600.237540461"
Jul 28 12:45:12.343366 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:12.343328    2576 generic.go:356] "Generic (PLEG): container finished" podID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerID="58b5a94fa7b75407bcf1b820e5305b3f5ef3c6855144efa78840a037738593a5" exitCode=0
Jul 28 12:45:12.343759 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:12.343411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-m7mhm" event={"ID":"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9","Type":"ContainerDied","Data":"58b5a94fa7b75407bcf1b820e5305b3f5ef3c6855144efa78840a037738593a5"}
Jul 28 12:45:13.488342 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.488318    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:13.604651 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.604600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqn7n\" (UniqueName: \"kubernetes.io/projected/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-kube-api-access-sqn7n\") pod \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") "
Jul 28 12:45:13.604790 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.604656    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-model-cache\") pod \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\" (UID: \"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9\") "
Jul 28 12:45:13.606588 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.606566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-kube-api-access-sqn7n" (OuterVolumeSpecName: "kube-api-access-sqn7n") pod "9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" (UID: "9c70dd70-b158-4f4f-a049-c0ba22f5e2f9"). InnerVolumeSpecName "kube-api-access-sqn7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:45:13.628198 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.628173    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" (UID: "9c70dd70-b158-4f4f-a049-c0ba22f5e2f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:45:13.705669 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.705647    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqn7n\" (UniqueName: \"kubernetes.io/projected/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-kube-api-access-sqn7n\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:45:13.705669 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:13.705667    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c70dd70-b158-4f4f-a049-c0ba22f5e2f9-model-cache\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:45:14.353944 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:14.353917    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-m7mhm"
Jul 28 12:45:14.354071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:14.353957    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-m7mhm" event={"ID":"9c70dd70-b158-4f4f-a049-c0ba22f5e2f9","Type":"ContainerDied","Data":"f266959d75ab2d48b9c216eb9becace611d37501feded2dc3a68f8392e386117"}
Jul 28 12:45:14.354071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:14.354002    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f266959d75ab2d48b9c216eb9becace611d37501feded2dc3a68f8392e386117"
Jul 28 12:45:18.319171 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.319141    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-pbms6"]
Jul 28 12:45:18.321573 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.319500    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerName="download-hf-model"
Jul 28 12:45:18.321573 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.319514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerName="download-hf-model"
Jul 28 12:45:18.321573 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.319534    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerName="s3-tls-init-custom"
Jul 28 12:45:18.321573 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.319539    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerName="s3-tls-init-custom"
Jul 28 12:45:18.321573 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.319588    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c70dd70-b158-4f4f-a049-c0ba22f5e2f9" containerName="s3-tls-init-custom"
Jul 28 12:45:18.322448 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.322430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.325207 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.325179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 12:45:18.325346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.325295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v9v8l\""
Jul 28 12:45:18.332164 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.332137    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-pbms6"]
Jul 28 12:45:18.439131 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.439104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15e9a40-e555-48bb-ae19-1045f0096d67-model-cache\") pod \"s3-tls-init-serving-pbms6\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") " pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.439246 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.439138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sx6n\" (UniqueName: \"kubernetes.io/projected/f15e9a40-e555-48bb-ae19-1045f0096d67-kube-api-access-8sx6n\") pod \"s3-tls-init-serving-pbms6\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") " pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.539739 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.539716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15e9a40-e555-48bb-ae19-1045f0096d67-model-cache\") pod \"s3-tls-init-serving-pbms6\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") " pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.539847 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.539747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sx6n\" (UniqueName: \"kubernetes.io/projected/f15e9a40-e555-48bb-ae19-1045f0096d67-kube-api-access-8sx6n\") pod \"s3-tls-init-serving-pbms6\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") " pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.540085 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.540065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15e9a40-e555-48bb-ae19-1045f0096d67-model-cache\") pod \"s3-tls-init-serving-pbms6\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") " pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.548310 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.548286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sx6n\" (UniqueName: \"kubernetes.io/projected/f15e9a40-e555-48bb-ae19-1045f0096d67-kube-api-access-8sx6n\") pod \"s3-tls-init-serving-pbms6\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") " pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.642052 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.642028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:18.775899 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:18.775877    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-pbms6"]
Jul 28 12:45:18.778416 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:45:18.778390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf15e9a40_e555_48bb_ae19_1045f0096d67.slice/crio-ac88ee40e3fa20cb8fbefb4d650ced707b2ac6ad51d1f0231286d8f6ae87940f WatchSource:0}: Error finding container ac88ee40e3fa20cb8fbefb4d650ced707b2ac6ad51d1f0231286d8f6ae87940f: Status 404 returned error can't find the container with id ac88ee40e3fa20cb8fbefb4d650ced707b2ac6ad51d1f0231286d8f6ae87940f
Jul 28 12:45:19.378402 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:19.378354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pbms6" event={"ID":"f15e9a40-e555-48bb-ae19-1045f0096d67","Type":"ContainerStarted","Data":"d897e78c6c5d25f739624d404ce62f8b3b7bbfc1157c54a858820433587378da"}
Jul 28 12:45:19.378752 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:19.378408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pbms6" event={"ID":"f15e9a40-e555-48bb-ae19-1045f0096d67","Type":"ContainerStarted","Data":"ac88ee40e3fa20cb8fbefb4d650ced707b2ac6ad51d1f0231286d8f6ae87940f"}
Jul 28 12:45:21.389226 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:21.389192    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerID="d897e78c6c5d25f739624d404ce62f8b3b7bbfc1157c54a858820433587378da" exitCode=0
Jul 28 12:45:21.389633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:21.389279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pbms6" event={"ID":"f15e9a40-e555-48bb-ae19-1045f0096d67","Type":"ContainerDied","Data":"d897e78c6c5d25f739624d404ce62f8b3b7bbfc1157c54a858820433587378da"}
Jul 28 12:45:22.395028 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:22.394994    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pbms6" event={"ID":"f15e9a40-e555-48bb-ae19-1045f0096d67","Type":"ContainerStarted","Data":"e556ffbcef42a383c93ef881d42ec92f282ccc0cebd2a42a3f8ae9b03227a8e5"}
Jul 28 12:45:22.412504 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:22.412458    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-pbms6" podStartSLOduration=2.762077869 podStartE2EDuration="4.412443879s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.65036601s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:45:22.411608808 +0000 UTC m=+615.312753200" watchObservedRunningTime="2026-07-28 12:45:22.412443879 +0000 UTC m=+615.313588269"
Jul 28 12:45:28.421902 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:28.421869    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerID="e556ffbcef42a383c93ef881d42ec92f282ccc0cebd2a42a3f8ae9b03227a8e5" exitCode=0
Jul 28 12:45:28.422246 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:28.421947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pbms6" event={"ID":"f15e9a40-e555-48bb-ae19-1045f0096d67","Type":"ContainerDied","Data":"e556ffbcef42a383c93ef881d42ec92f282ccc0cebd2a42a3f8ae9b03227a8e5"}
Jul 28 12:45:29.559336 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.559314    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:29.728861 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.728802    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sx6n\" (UniqueName: \"kubernetes.io/projected/f15e9a40-e555-48bb-ae19-1045f0096d67-kube-api-access-8sx6n\") pod \"f15e9a40-e555-48bb-ae19-1045f0096d67\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") "
Jul 28 12:45:29.728861 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.728861    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15e9a40-e555-48bb-ae19-1045f0096d67-model-cache\") pod \"f15e9a40-e555-48bb-ae19-1045f0096d67\" (UID: \"f15e9a40-e555-48bb-ae19-1045f0096d67\") "
Jul 28 12:45:29.730563 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.730537    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15e9a40-e555-48bb-ae19-1045f0096d67-kube-api-access-8sx6n" (OuterVolumeSpecName: "kube-api-access-8sx6n") pod "f15e9a40-e555-48bb-ae19-1045f0096d67" (UID: "f15e9a40-e555-48bb-ae19-1045f0096d67"). InnerVolumeSpecName "kube-api-access-8sx6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:45:29.751154 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.751130    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15e9a40-e555-48bb-ae19-1045f0096d67-model-cache" (OuterVolumeSpecName: "model-cache") pod "f15e9a40-e555-48bb-ae19-1045f0096d67" (UID: "f15e9a40-e555-48bb-ae19-1045f0096d67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:45:29.830157 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.830130    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15e9a40-e555-48bb-ae19-1045f0096d67-model-cache\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:45:29.830157 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:29.830155    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sx6n\" (UniqueName: \"kubernetes.io/projected/f15e9a40-e555-48bb-ae19-1045f0096d67-kube-api-access-8sx6n\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:45:30.432046 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:30.432024    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-pbms6"
Jul 28 12:45:30.432177 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:30.432037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-pbms6" event={"ID":"f15e9a40-e555-48bb-ae19-1045f0096d67","Type":"ContainerDied","Data":"ac88ee40e3fa20cb8fbefb4d650ced707b2ac6ad51d1f0231286d8f6ae87940f"}
Jul 28 12:45:30.432177 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:30.432076    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac88ee40e3fa20cb8fbefb4d650ced707b2ac6ad51d1f0231286d8f6ae87940f"
Jul 28 12:45:44.212019 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.211988    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"]
Jul 28 12:45:44.212594 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.212346    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerName="s3-tls-init-serving"
Jul 28 12:45:44.212594 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.212358    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerName="s3-tls-init-serving"
Jul 28 12:45:44.212594 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.212383    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerName="download-hf-model"
Jul 28 12:45:44.212594 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.212392    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerName="download-hf-model"
Jul 28 12:45:44.212594 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.212468    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f15e9a40-e555-48bb-ae19-1045f0096d67" containerName="s3-tls-init-serving"
Jul 28 12:45:44.216092 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.216074    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.218920 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.218899    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:45:44.219027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.218963    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:45:44.219027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.218997    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 12:45:44.219027 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.218998    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:45:44.219183 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.219025    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:45:44.219257 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.219247    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 12:45:44.219312 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.219278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 12:45:44.229791 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.229765    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"]
Jul 28 12:45:44.235179 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.235152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgvm\" (UniqueName: \"kubernetes.io/projected/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-api-access-xrgvm\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.235302 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.235222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.235302 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.235285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.235444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.235313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.235444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.235334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f15a80bc-e9ff-4425-936f-8f782827fcdf-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336238 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrgvm\" (UniqueName: \"kubernetes.io/projected/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-api-access-xrgvm\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f15a80bc-e9ff-4425-936f-8f782827fcdf-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336558 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:45:44.336419    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 28 12:45:44.336558 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:45:44.336492    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls podName:f15a80bc-e9ff-4425-936f-8f782827fcdf nodeName:}" failed. No retries permitted until 2026-07-28 12:45:44.836472421 +0000 UTC m=+637.737616789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls") pod "isvc-sklearn-batcher-predictor-5576655544-65fr7" (UID: "f15a80bc-e9ff-4425-936f-8f782827fcdf") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 28 12:45:44.336793 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336774    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f15a80bc-e9ff-4425-936f-8f782827fcdf-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.336995 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.337040 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.336997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.347310 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.347289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrgvm\" (UniqueName: \"kubernetes.io/projected/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-api-access-xrgvm\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.839488 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.839459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:44.841668 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:44.841642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5576655544-65fr7\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:45.127640 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:45.127574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:45:45.263360 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:45.263336    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"]
Jul 28 12:45:45.265342 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:45:45.265317    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15a80bc_e9ff_4425_936f_8f782827fcdf.slice/crio-bd02126f139aef6429e101469fef3b5c2d422daca2c44cc9d580ec01a8fb280e WatchSource:0}: Error finding container bd02126f139aef6429e101469fef3b5c2d422daca2c44cc9d580ec01a8fb280e: Status 404 returned error can't find the container with id bd02126f139aef6429e101469fef3b5c2d422daca2c44cc9d580ec01a8fb280e
Jul 28 12:45:45.497234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:45.497159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerStarted","Data":"34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd"}
Jul 28 12:45:45.497234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:45.497226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerStarted","Data":"bd02126f139aef6429e101469fef3b5c2d422daca2c44cc9d580ec01a8fb280e"}
Jul 28 12:45:49.514410 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:49.514361    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerID="34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd" exitCode=0
Jul 28 12:45:49.514746 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:49.514437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerDied","Data":"34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd"}
Jul 28 12:45:49.515650 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:45:49.515636    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:46:02.598715 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:02.598676    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerStarted","Data":"21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63"}
Jul 28 12:46:04.608663 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:04.608627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerStarted","Data":"04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302"}
Jul 28 12:46:06.618839 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:06.618804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerStarted","Data":"4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd"}
Jul 28 12:46:06.619196 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:06.618879    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:46:06.619196 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:06.618914    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:46:06.620332 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:06.620301    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:06.643466 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:06.643422    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podStartSLOduration=2.049084401 podStartE2EDuration="22.643408758s" totalImagesPullingTime="16.865008208s" totalInitContainerRuntime="3.729316149s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:45:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:46:06.642574235 +0000 UTC m=+659.543718624" watchObservedRunningTime="2026-07-28 12:46:06.643408758 +0000 UTC m=+659.544553146"
Jul 28 12:46:07.622805 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:07.622766    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:46:07.623134 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:07.622808    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:07.623909 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:07.623883    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:08.626944 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:08.626903    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:08.627346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:08.627210    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:13.632252 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:13.632223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:46:13.632801 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:13.632766    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:13.633167 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:13.633144    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:23.633453 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:23.633402    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:23.633870 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:23.633780    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:33.632914 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:33.632879    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:33.633390 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:33.633316    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:43.632948 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:43.632902    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:43.633443 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:43.633411    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:46:53.633410 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:53.633353    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:46:53.633956 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:46:53.633926    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:03.633007 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:03.632963    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:03.633596 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:03.633567    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:13.633590 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:13.633555    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:13.634189 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:13.633661    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:19.422610 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.422577    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"]
Jul 28 12:47:19.426439 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.426422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.429387 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.429357    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 12:47:19.429501 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.429362    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 12:47:19.439403 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.439382    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"]
Jul 28 12:47:19.439618 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.439584    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" containerID="cri-o://21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63" gracePeriod=30
Jul 28 12:47:19.439691 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.439658    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" containerID="cri-o://04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302" gracePeriod=30
Jul 28 12:47:19.439755 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.439652    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" containerID="cri-o://4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd" gracePeriod=30
Jul 28 12:47:19.442808 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.442786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.443144 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.443120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knppq\" (UniqueName: \"kubernetes.io/projected/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-api-access-knppq\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.443235 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.443207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/399fa73d-933c-4412-8b4f-9bd4e09ce52b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.443235 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.443230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.443345 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.443263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.445192 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.445150    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"]
Jul 28 12:47:19.543638 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.543615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/399fa73d-933c-4412-8b4f-9bd4e09ce52b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.543776 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.543645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.543776 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.543671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.543776 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.543745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.543942 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.543778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knppq\" (UniqueName: \"kubernetes.io/projected/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-api-access-knppq\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.544407 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.544336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.544407 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.544360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.544998 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.544975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.546520 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.546501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/399fa73d-933c-4412-8b4f-9bd4e09ce52b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.554062 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.554037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knppq\" (UniqueName: \"kubernetes.io/projected/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-api-access-knppq\") pod \"isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.735848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.735789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:19.863099 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.863075    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"]
Jul 28 12:47:19.864485 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:47:19.864461    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399fa73d_933c_4412_8b4f_9bd4e09ce52b.slice/crio-4784c4b515d7a1872ec658e4db411fdc4351acee3a819338315fb33e28af4732 WatchSource:0}: Error finding container 4784c4b515d7a1872ec658e4db411fdc4351acee3a819338315fb33e28af4732: Status 404 returned error can't find the container with id 4784c4b515d7a1872ec658e4db411fdc4351acee3a819338315fb33e28af4732
Jul 28 12:47:19.925685 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.925661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerStarted","Data":"e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31"}
Jul 28 12:47:19.925788 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.925734    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerStarted","Data":"4784c4b515d7a1872ec658e4db411fdc4351acee3a819338315fb33e28af4732"}
Jul 28 12:47:19.928041 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.928023    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerID="04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302" exitCode=2
Jul 28 12:47:19.928131 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:19.928047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerDied","Data":"04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302"}
Jul 28 12:47:23.627936 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.627898    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:47:23.633299 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.633274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:23.633665 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.633635    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:23.948856 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.948790    2576 generic.go:356] "Generic (PLEG): container finished" podID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerID="e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31" exitCode=0
Jul 28 12:47:23.948976 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.948873    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerDied","Data":"e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31"}
Jul 28 12:47:23.951955 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.951923    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerID="21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63" exitCode=0
Jul 28 12:47:23.952076 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:23.951987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerDied","Data":"21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63"}
Jul 28 12:47:24.958655 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.958623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerStarted","Data":"8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716"}
Jul 28 12:47:24.959001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.958686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerStarted","Data":"330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0"}
Jul 28 12:47:24.959001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.958713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerStarted","Data":"7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12"}
Jul 28 12:47:24.959001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.958979    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:24.959136 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.959035    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:24.960184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.960159    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:47:24.983602 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:24.983563    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podStartSLOduration=2.268872819 podStartE2EDuration="5.983550567s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714677748s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:47:24.982355402 +0000 UTC m=+737.883499791" watchObservedRunningTime="2026-07-28 12:47:24.983550567 +0000 UTC m=+737.884694955"
Jul 28 12:47:25.963002 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:25.962971    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:47:25.963471 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:25.963040    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:25.964166 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:25.964140    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:26.967581 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:26.967545    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:47:26.968102 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:26.968081    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:28.627789 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:28.627753    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:47:31.971739 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:31.971704    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:47:31.972422 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:31.972365    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:47:31.972668 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:31.972647    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:33.627366 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:33.627317    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:47:33.627891 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:33.627525    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:33.632777 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:33.632736    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:33.633010 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:33.632991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:38.627969 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:38.627931    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:47:41.973010 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:41.972971    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:47:41.973488 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:41.973439    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:43.627877 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:43.627838    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:47:43.633299 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:43.633275    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 12:47:43.633436 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:43.633401    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:43.633712 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:43.633685    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:47:43.633812 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:43.633774    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:48.627015 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:48.626981    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 12:47:49.591351 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.591327    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:49.682529 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682504    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f15a80bc-e9ff-4425-936f-8f782827fcdf-kserve-provision-location\") pod \"f15a80bc-e9ff-4425-936f-8f782827fcdf\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") "
Jul 28 12:47:49.682931 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682540    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-rbac-proxy-sar-config\") pod \"f15a80bc-e9ff-4425-936f-8f782827fcdf\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") "
Jul 28 12:47:49.682931 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682586    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls\") pod \"f15a80bc-e9ff-4425-936f-8f782827fcdf\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") "
Jul 28 12:47:49.682931 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrgvm\" (UniqueName: \"kubernetes.io/projected/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-api-access-xrgvm\") pod \"f15a80bc-e9ff-4425-936f-8f782827fcdf\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") "
Jul 28 12:47:49.682931 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682697    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-cabundle-cert\") pod \"f15a80bc-e9ff-4425-936f-8f782827fcdf\" (UID: \"f15a80bc-e9ff-4425-936f-8f782827fcdf\") "
Jul 28 12:47:49.682931 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682874    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15a80bc-e9ff-4425-936f-8f782827fcdf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f15a80bc-e9ff-4425-936f-8f782827fcdf" (UID: "f15a80bc-e9ff-4425-936f-8f782827fcdf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:47:49.683151 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.682956    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f15a80bc-e9ff-4425-936f-8f782827fcdf" (UID: "f15a80bc-e9ff-4425-936f-8f782827fcdf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:47:49.683409 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.683301    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f15a80bc-e9ff-4425-936f-8f782827fcdf" (UID: "f15a80bc-e9ff-4425-936f-8f782827fcdf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:47:49.684819 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.684794    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f15a80bc-e9ff-4425-936f-8f782827fcdf" (UID: "f15a80bc-e9ff-4425-936f-8f782827fcdf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:47:49.685278 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.685261    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-api-access-xrgvm" (OuterVolumeSpecName: "kube-api-access-xrgvm") pod "f15a80bc-e9ff-4425-936f-8f782827fcdf" (UID: "f15a80bc-e9ff-4425-936f-8f782827fcdf"). InnerVolumeSpecName "kube-api-access-xrgvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:47:49.784036 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.784012    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrgvm\" (UniqueName: \"kubernetes.io/projected/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-api-access-xrgvm\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:47:49.784130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.784038    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:47:49.784130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.784051    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f15a80bc-e9ff-4425-936f-8f782827fcdf-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:47:49.784130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.784065    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f15a80bc-e9ff-4425-936f-8f782827fcdf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:47:49.784130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:49.784077    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f15a80bc-e9ff-4425-936f-8f782827fcdf-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:47:50.072918 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.072849    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerID="4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd" exitCode=0
Jul 28 12:47:50.072918 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.072886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerDied","Data":"4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd"}
Jul 28 12:47:50.072918 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.072916    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"
Jul 28 12:47:50.073132 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.072956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7" event={"ID":"f15a80bc-e9ff-4425-936f-8f782827fcdf","Type":"ContainerDied","Data":"bd02126f139aef6429e101469fef3b5c2d422daca2c44cc9d580ec01a8fb280e"}
Jul 28 12:47:50.073132 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.073007    2576 scope.go:117] "RemoveContainer" containerID="4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd"
Jul 28 12:47:50.081429 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.081410    2576 scope.go:117] "RemoveContainer" containerID="04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302"
Jul 28 12:47:50.088687 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.088665    2576 scope.go:117] "RemoveContainer" containerID="21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63"
Jul 28 12:47:50.093832 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.093814    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"]
Jul 28 12:47:50.097038 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.097023    2576 scope.go:117] "RemoveContainer" containerID="34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd"
Jul 28 12:47:50.097816 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.097795    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5576655544-65fr7"]
Jul 28 12:47:50.104164 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104148    2576 scope.go:117] "RemoveContainer" containerID="4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd"
Jul 28 12:47:50.104444 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:47:50.104419    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd\": container with ID starting with 4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd not found: ID does not exist" containerID="4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd"
Jul 28 12:47:50.104511 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104462    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd"} err="failed to get container status \"4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd\": rpc error: code = NotFound desc = could not find container \"4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd\": container with ID starting with 4b08f2929ba3eccfe9c7a70fa0e836f64020c69cbd1fce05f11f32e92ecc39bd not found: ID does not exist"
Jul 28 12:47:50.104511 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104484    2576 scope.go:117] "RemoveContainer" containerID="04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302"
Jul 28 12:47:50.104677 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:47:50.104662    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302\": container with ID starting with 04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302 not found: ID does not exist" containerID="04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302"
Jul 28 12:47:50.104731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104679    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302"} err="failed to get container status \"04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302\": rpc error: code = NotFound desc = could not find container \"04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302\": container with ID starting with 04f56375d5a047c40fbb6aafe3ec6ce790909bbeb24ca6a0d223980babb3e302 not found: ID does not exist"
Jul 28 12:47:50.104731 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104691    2576 scope.go:117] "RemoveContainer" containerID="21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63"
Jul 28 12:47:50.104854 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:47:50.104840    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63\": container with ID starting with 21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63 not found: ID does not exist" containerID="21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63"
Jul 28 12:47:50.104899 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104855    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63"} err="failed to get container status \"21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63\": rpc error: code = NotFound desc = could not find container \"21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63\": container with ID starting with 21540ce9108e38715eb1e681a40c484f54c07f09ab3fbb2cd83acd0188e6cb63 not found: ID does not exist"
Jul 28 12:47:50.104899 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.104874    2576 scope.go:117] "RemoveContainer" containerID="34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd"
Jul 28 12:47:50.105064 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:47:50.105046    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd\": container with ID starting with 34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd not found: ID does not exist" containerID="34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd"
Jul 28 12:47:50.105104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:50.105067    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd"} err="failed to get container status \"34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd\": rpc error: code = NotFound desc = could not find container \"34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd\": container with ID starting with 34b6d62380ed8624a89462ffa642c0a9c34645363b6ec2ec5a8a426e4a0cd3bd not found: ID does not exist"
Jul 28 12:47:51.721140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:51.721108    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" path="/var/lib/kubelet/pods/f15a80bc-e9ff-4425-936f-8f782827fcdf/volumes"
Jul 28 12:47:51.972910 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:51.972819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:47:51.973249 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:47:51.973229    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:01.972715 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:01.972662    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:48:01.973120 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:01.973081    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:11.972564 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:11.972528    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:48:11.973095 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:11.973069    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:21.972835 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:21.972787    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:48:21.973303 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:21.973237    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:31.972511 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:31.972422    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:48:31.972917 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:31.972893    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:48:41.973111 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:41.973084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:48:41.973553 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:41.973215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:48:54.594817 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.594783    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"]
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595198    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595212    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595223    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595229    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595237    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="storage-initializer"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595244    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="storage-initializer"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595256    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595261    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595317    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kserve-container"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595328    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="agent"
Jul 28 12:48:54.595361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.595333    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f15a80bc-e9ff-4425-936f-8f782827fcdf" containerName="kube-rbac-proxy"
Jul 28 12:48:54.598558 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.598541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.601413 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.601388    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 12:48:54.601509 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.601430    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 12:48:54.610097 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.610044    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"]
Jul 28 12:48:54.612725 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.612703    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"]
Jul 28 12:48:54.613003 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.612958    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" containerID="cri-o://7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12" gracePeriod=30
Jul 28 12:48:54.613112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.613013    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" containerID="cri-o://8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716" gracePeriod=30
Jul 28 12:48:54.613215 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.613167    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" containerID="cri-o://330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0" gracePeriod=30
Jul 28 12:48:54.714145 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.714115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.714282 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.714164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50444542-9319-4865-9ee1-b08ea60984af-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.714282 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.714254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krk5g\" (UniqueName: \"kubernetes.io/projected/50444542-9319-4865-9ee1-b08ea60984af-kube-api-access-krk5g\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.815243 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.815212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50444542-9319-4865-9ee1-b08ea60984af-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.815462 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.815278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krk5g\" (UniqueName: \"kubernetes.io/projected/50444542-9319-4865-9ee1-b08ea60984af-kube-api-access-krk5g\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.815462 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.815392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.815622 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:48:54.815600    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 28 12:48:54.815691 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:48:54.815656    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls podName:50444542-9319-4865-9ee1-b08ea60984af nodeName:}" failed. No retries permitted until 2026-07-28 12:48:55.31564026 +0000 UTC m=+828.216784633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-vv8rb" (UID: "50444542-9319-4865-9ee1-b08ea60984af") : secret "message-dumper-predictor-serving-cert" not found
Jul 28 12:48:54.816000 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.815981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50444542-9319-4865-9ee1-b08ea60984af-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:54.823908 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:54.823888    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krk5g\" (UniqueName: \"kubernetes.io/projected/50444542-9319-4865-9ee1-b08ea60984af-kube-api-access-krk5g\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:55.319602 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:55.319571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:55.322008 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:55.321973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-vv8rb\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:55.364438 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:55.364410    2576 generic.go:356] "Generic (PLEG): container finished" podID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerID="330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0" exitCode=2
Jul 28 12:48:55.364601 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:55.364483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerDied","Data":"330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0"}
Jul 28 12:48:55.509422 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:55.509389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:55.636249 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:55.636225    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"]
Jul 28 12:48:55.638315 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:48:55.638291    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50444542_9319_4865_9ee1_b08ea60984af.slice/crio-3f665d96193019c0c9dac0d7564dce0b13a652ea836518b4408d068c660a5bc4 WatchSource:0}: Error finding container 3f665d96193019c0c9dac0d7564dce0b13a652ea836518b4408d068c660a5bc4: Status 404 returned error can't find the container with id 3f665d96193019c0c9dac0d7564dce0b13a652ea836518b4408d068c660a5bc4
Jul 28 12:48:56.369627 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:56.369595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" event={"ID":"50444542-9319-4865-9ee1-b08ea60984af","Type":"ContainerStarted","Data":"3f665d96193019c0c9dac0d7564dce0b13a652ea836518b4408d068c660a5bc4"}
Jul 28 12:48:56.968494 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:56.968462    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:48:57.376089 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:57.376058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" event={"ID":"50444542-9319-4865-9ee1-b08ea60984af","Type":"ContainerStarted","Data":"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"}
Jul 28 12:48:57.376283 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:57.376111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" event={"ID":"50444542-9319-4865-9ee1-b08ea60984af","Type":"ContainerStarted","Data":"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"}
Jul 28 12:48:57.376283 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:57.376201    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:57.395208 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:57.395157    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" podStartSLOduration=2.151114109 podStartE2EDuration="3.395140808s" totalImagesPullingTime="1.244026699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:48:57.395010432 +0000 UTC m=+830.296154822" watchObservedRunningTime="2026-07-28 12:48:57.395140808 +0000 UTC m=+830.296285198"
Jul 28 12:48:58.381434 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:58.381401    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:58.382927 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:58.382908    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:48:59.388383 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:59.388345    2576 generic.go:356] "Generic (PLEG): container finished" podID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerID="7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12" exitCode=0
Jul 28 12:48:59.388744 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:48:59.388399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerDied","Data":"7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12"}
Jul 28 12:49:01.968585 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:01.968538    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:49:01.972969 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:01.972928    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:49:01.973281 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:01.973251    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:05.398666 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:05.398632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:49:06.967884 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:06.967842    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:49:06.968348 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:06.967970    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:49:11.967960 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:11.967917    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:49:11.972284 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:11.972247    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:49:11.972621 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:11.972597    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:14.602700 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.602663    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"]
Jul 28 12:49:14.611547 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.611521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.614310 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.614287    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 12:49:14.614310 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.614298    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 12:49:14.615568 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.615547    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"]
Jul 28 12:49:14.674765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.674729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb8d456-34d6-4c1a-af28-76142161d439-kserve-provision-location\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.674765 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.674764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9fz\" (UniqueName: \"kubernetes.io/projected/3bb8d456-34d6-4c1a-af28-76142161d439-kube-api-access-nb9fz\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.674926 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.674790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-cabundle-cert\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.674926 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.674885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.674992 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.674931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.776198 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.776169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.776361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.776221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.776361 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:49:14.776310    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 28 12:49:14.776537 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:49:14.776396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls podName:3bb8d456-34d6-4c1a-af28-76142161d439 nodeName:}" failed. No retries permitted until 2026-07-28 12:49:15.276352291 +0000 UTC m=+848.177496663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls") pod "isvc-logger-predictor-65bf8d84bc-mvrkt" (UID: "3bb8d456-34d6-4c1a-af28-76142161d439") : secret "isvc-logger-predictor-serving-cert" not found
Jul 28 12:49:14.776609 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.776539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb8d456-34d6-4c1a-af28-76142161d439-kserve-provision-location\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.776609 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.776600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9fz\" (UniqueName: \"kubernetes.io/projected/3bb8d456-34d6-4c1a-af28-76142161d439-kube-api-access-nb9fz\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.776751 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.776637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-cabundle-cert\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.777026 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.777003    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb8d456-34d6-4c1a-af28-76142161d439-kserve-provision-location\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.777114 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.777031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.777216 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.777200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-cabundle-cert\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:14.784605 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:14.784588    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9fz\" (UniqueName: \"kubernetes.io/projected/3bb8d456-34d6-4c1a-af28-76142161d439-kube-api-access-nb9fz\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:15.279869 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:15.279839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:15.282298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:15.282273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls\") pod \"isvc-logger-predictor-65bf8d84bc-mvrkt\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:15.523651 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:15.523619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:15.653140 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:15.653116    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"]
Jul 28 12:49:15.654403 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:49:15.654354    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb8d456_34d6_4c1a_af28_76142161d439.slice/crio-23c85ca05d0cbf7376011b8675ea212405b5b49177715c6bacecc9aeb2ee32ad WatchSource:0}: Error finding container 23c85ca05d0cbf7376011b8675ea212405b5b49177715c6bacecc9aeb2ee32ad: Status 404 returned error can't find the container with id 23c85ca05d0cbf7376011b8675ea212405b5b49177715c6bacecc9aeb2ee32ad
Jul 28 12:49:16.472998 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:16.472960    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerStarted","Data":"e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996"}
Jul 28 12:49:16.473192 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:16.473034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerStarted","Data":"23c85ca05d0cbf7376011b8675ea212405b5b49177715c6bacecc9aeb2ee32ad"}
Jul 28 12:49:16.968639 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:16.968599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:49:20.493484 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:20.493448    2576 generic.go:356] "Generic (PLEG): container finished" podID="3bb8d456-34d6-4c1a-af28-76142161d439" containerID="e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996" exitCode=0
Jul 28 12:49:20.493964 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:20.493528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerDied","Data":"e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996"}
Jul 28 12:49:21.501384 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.501335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerStarted","Data":"a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58"}
Jul 28 12:49:21.501737 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.501426    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerStarted","Data":"7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6"}
Jul 28 12:49:21.501737 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.501457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerStarted","Data":"e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905"}
Jul 28 12:49:21.501838 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.501762    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:21.501931 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.501913    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:21.503314 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.503289    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:21.524795 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.524738    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podStartSLOduration=3.727930091 podStartE2EDuration="7.524725413s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796795322s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:49:21.523362191 +0000 UTC m=+854.424506625" watchObservedRunningTime="2026-07-28 12:49:21.524725413 +0000 UTC m=+854.425869805"
Jul 28 12:49:21.968648 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.968602    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 12:49:21.972984 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.972944    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 28 12:49:21.973147 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.973062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:49:21.973314 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.973295    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:21.973426 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:21.973413    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:49:22.508710 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:22.508677    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:22.509184 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:22.509074    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:22.509962 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:22.509934    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:23.513006 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:23.512958    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:23.513429 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:23.513281    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:24.766624 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.766602    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:49:24.858463 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-rbac-proxy-sar-config\") pod \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") "
Jul 28 12:49:24.858597 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858473    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knppq\" (UniqueName: \"kubernetes.io/projected/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-api-access-knppq\") pod \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") "
Jul 28 12:49:24.858597 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858515    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kserve-provision-location\") pod \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") "
Jul 28 12:49:24.858597 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858572    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/399fa73d-933c-4412-8b4f-9bd4e09ce52b-proxy-tls\") pod \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") "
Jul 28 12:49:24.858746 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858611    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-cabundle-cert\") pod \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\" (UID: \"399fa73d-933c-4412-8b4f-9bd4e09ce52b\") "
Jul 28 12:49:24.858800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858779    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "399fa73d-933c-4412-8b4f-9bd4e09ce52b" (UID: "399fa73d-933c-4412-8b4f-9bd4e09ce52b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:49:24.858857 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858812    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "399fa73d-933c-4412-8b4f-9bd4e09ce52b" (UID: "399fa73d-933c-4412-8b4f-9bd4e09ce52b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:49:24.858940 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858922    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:49:24.859000 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858942    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:49:24.859043 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.858996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "399fa73d-933c-4412-8b4f-9bd4e09ce52b" (UID: "399fa73d-933c-4412-8b4f-9bd4e09ce52b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:49:24.860615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.860597    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-api-access-knppq" (OuterVolumeSpecName: "kube-api-access-knppq") pod "399fa73d-933c-4412-8b4f-9bd4e09ce52b" (UID: "399fa73d-933c-4412-8b4f-9bd4e09ce52b"). InnerVolumeSpecName "kube-api-access-knppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:49:24.860720 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.860700    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399fa73d-933c-4412-8b4f-9bd4e09ce52b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "399fa73d-933c-4412-8b4f-9bd4e09ce52b" (UID: "399fa73d-933c-4412-8b4f-9bd4e09ce52b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:49:24.959468 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.959408    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/399fa73d-933c-4412-8b4f-9bd4e09ce52b-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:49:24.959468 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.959428    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/399fa73d-933c-4412-8b4f-9bd4e09ce52b-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:49:24.959468 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:24.959437    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knppq\" (UniqueName: \"kubernetes.io/projected/399fa73d-933c-4412-8b4f-9bd4e09ce52b-kube-api-access-knppq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:49:25.524389 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.524346    2576 generic.go:356] "Generic (PLEG): container finished" podID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerID="8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716" exitCode=0
Jul 28 12:49:25.524558 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.524401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerDied","Data":"8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716"}
Jul 28 12:49:25.524558 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.524428    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"
Jul 28 12:49:25.524558 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.524464    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk" event={"ID":"399fa73d-933c-4412-8b4f-9bd4e09ce52b","Type":"ContainerDied","Data":"4784c4b515d7a1872ec658e4db411fdc4351acee3a819338315fb33e28af4732"}
Jul 28 12:49:25.524558 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.524524    2576 scope.go:117] "RemoveContainer" containerID="8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716"
Jul 28 12:49:25.540387 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.540355    2576 scope.go:117] "RemoveContainer" containerID="330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0"
Jul 28 12:49:25.547958 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.547938    2576 scope.go:117] "RemoveContainer" containerID="7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12"
Jul 28 12:49:25.551166 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.551144    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"]
Jul 28 12:49:25.555025 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.554996    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5bd4978c59-mcngk"]
Jul 28 12:49:25.556227 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.556215    2576 scope.go:117] "RemoveContainer" containerID="e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31"
Jul 28 12:49:25.563563 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.563549    2576 scope.go:117] "RemoveContainer" containerID="8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716"
Jul 28 12:49:25.563799 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:49:25.563781    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716\": container with ID starting with 8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716 not found: ID does not exist" containerID="8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716"
Jul 28 12:49:25.563885 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.563810    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716"} err="failed to get container status \"8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716\": rpc error: code = NotFound desc = could not find container \"8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716\": container with ID starting with 8e7f9fdd94ea6aa0b6dde93451878fcb617ada661c1ace4cdf7e89b52951b716 not found: ID does not exist"
Jul 28 12:49:25.563885 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.563832    2576 scope.go:117] "RemoveContainer" containerID="330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0"
Jul 28 12:49:25.564088 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:49:25.564072    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0\": container with ID starting with 330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0 not found: ID does not exist" containerID="330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0"
Jul 28 12:49:25.564127 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.564093    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0"} err="failed to get container status \"330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0\": rpc error: code = NotFound desc = could not find container \"330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0\": container with ID starting with 330cce7a33ebc6146abf11df454cf897eb5d04f8511e90a7358eb9ed491656e0 not found: ID does not exist"
Jul 28 12:49:25.564127 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.564108    2576 scope.go:117] "RemoveContainer" containerID="7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12"
Jul 28 12:49:25.564319 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:49:25.564300    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12\": container with ID starting with 7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12 not found: ID does not exist" containerID="7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12"
Jul 28 12:49:25.564382 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.564326    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12"} err="failed to get container status \"7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12\": rpc error: code = NotFound desc = could not find container \"7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12\": container with ID starting with 7830637f810ccb2effce181a7f1a72024d50ac41a2f5962dafa9ce46f1736f12 not found: ID does not exist"
Jul 28 12:49:25.564382 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.564340    2576 scope.go:117] "RemoveContainer" containerID="e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31"
Jul 28 12:49:25.564587 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:49:25.564571    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31\": container with ID starting with e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31 not found: ID does not exist" containerID="e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31"
Jul 28 12:49:25.564640 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.564595    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31"} err="failed to get container status \"e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31\": rpc error: code = NotFound desc = could not find container \"e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31\": container with ID starting with e1c05bca9babd3bf798d01cd4df54f71f11c746d94796ea3c34ddea9cf336a31 not found: ID does not exist"
Jul 28 12:49:25.720352 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:25.720320    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" path="/var/lib/kubelet/pods/399fa73d-933c-4412-8b4f-9bd4e09ce52b/volumes"
Jul 28 12:49:28.517953 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:28.517917    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:49:28.518629 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:28.518587    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:28.518996 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:28.518975    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:38.519010 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:38.518967    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:38.519635 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:38.519613    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:48.524115 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:48.524065    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:48.524584 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:48.524538    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:49:58.518973 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:58.518928    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:49:58.519474 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:49:58.519385    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:50:08.518791 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:08.518746    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:50:08.519269 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:08.519186    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:50:18.518555 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:18.518493    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:50:18.520878 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:18.518900    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:50:28.518967 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:28.518927    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:50:28.519457 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:28.519419    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:50:38.519350 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:38.519319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:50:38.519908 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:38.519510    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:50:49.662917 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.662886    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-vv8rb_50444542-9319-4865-9ee1-b08ea60984af/kserve-container/0.log"
Jul 28 12:50:49.891462 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.891434    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"]
Jul 28 12:50:49.891750 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.891690    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" containerID="cri-o://e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905" gracePeriod=30
Jul 28 12:50:49.891890 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.891723    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" containerID="cri-o://7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6" gracePeriod=30
Jul 28 12:50:49.891890 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.891713    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" containerID="cri-o://a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58" gracePeriod=30
Jul 28 12:50:49.938188 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938128    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"]
Jul 28 12:50:49.938634 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938618    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="storage-initializer"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938636    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="storage-initializer"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938645    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938650    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938665    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938671    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938680    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy"
Jul 28 12:50:49.938688 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938685    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy"
Jul 28 12:50:49.938903 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938745    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kserve-container"
Jul 28 12:50:49.938903 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938753    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="kube-rbac-proxy"
Jul 28 12:50:49.938903 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.938758    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="399fa73d-933c-4412-8b4f-9bd4e09ce52b" containerName="agent"
Jul 28 12:50:49.941969 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.941956    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:49.945089 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.945068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 12:50:49.945185 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.945118    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 12:50:49.954615 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:49.954595    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"]
Jul 28 12:50:50.068818 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.068787    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"]
Jul 28 12:50:50.069043 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.069017    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kube-rbac-proxy" containerID="cri-o://dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf" gracePeriod=30
Jul 28 12:50:50.069160 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.069003    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kserve-container" containerID="cri-o://c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c" gracePeriod=30
Jul 28 12:50:50.083974 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.083953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ace1d49-beb1-45ef-80f6-0a4db1412621-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.084061 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.083984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ace1d49-beb1-45ef-80f6-0a4db1412621-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.084061 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.084004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.084129 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.084054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lcn\" (UniqueName: \"kubernetes.io/projected/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-api-access-h6lcn\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.084168 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.084155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.185463 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.185444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.185533 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.185487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lcn\" (UniqueName: \"kubernetes.io/projected/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-api-access-h6lcn\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.185614 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.185559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.185614 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.185598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ace1d49-beb1-45ef-80f6-0a4db1412621-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.185697 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.185630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ace1d49-beb1-45ef-80f6-0a4db1412621-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.186050 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.186031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ace1d49-beb1-45ef-80f6-0a4db1412621-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.186139 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.186120    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.186255 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.186240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.187835 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.187817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ace1d49-beb1-45ef-80f6-0a4db1412621-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.194658 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.194601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lcn\" (UniqueName: \"kubernetes.io/projected/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-api-access-h6lcn\") pod \"isvc-autogluon-v1-predictor-574b5b7495-lphsz\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.251527 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.251498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:50:50.311177 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.311071    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:50:50.383945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.383930    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"]
Jul 28 12:50:50.385966 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:50:50.385942    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ace1d49_beb1_45ef_80f6_0a4db1412621.slice/crio-f4ce669bd5c0b20fa5380baeeb9e9be56ab55c903173bbe2d8ec9572931e7d2a WatchSource:0}: Error finding container f4ce669bd5c0b20fa5380baeeb9e9be56ab55c903173bbe2d8ec9572931e7d2a: Status 404 returned error can't find the container with id f4ce669bd5c0b20fa5380baeeb9e9be56ab55c903173bbe2d8ec9572931e7d2a
Jul 28 12:50:50.387076 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.387057    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50444542-9319-4865-9ee1-b08ea60984af-kube-rbac-proxy-sar-config\") pod \"50444542-9319-4865-9ee1-b08ea60984af\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") "
Jul 28 12:50:50.387142 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.387094    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls\") pod \"50444542-9319-4865-9ee1-b08ea60984af\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") "
Jul 28 12:50:50.387196 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.387144    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krk5g\" (UniqueName: \"kubernetes.io/projected/50444542-9319-4865-9ee1-b08ea60984af-kube-api-access-krk5g\") pod \"50444542-9319-4865-9ee1-b08ea60984af\" (UID: \"50444542-9319-4865-9ee1-b08ea60984af\") "
Jul 28 12:50:50.387503 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.387444    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50444542-9319-4865-9ee1-b08ea60984af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50444542-9319-4865-9ee1-b08ea60984af" (UID: "50444542-9319-4865-9ee1-b08ea60984af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:50:50.389226 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.389196    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50444542-9319-4865-9ee1-b08ea60984af-kube-api-access-krk5g" (OuterVolumeSpecName: "kube-api-access-krk5g") pod "50444542-9319-4865-9ee1-b08ea60984af" (UID: "50444542-9319-4865-9ee1-b08ea60984af"). InnerVolumeSpecName "kube-api-access-krk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:50:50.389321 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.389302    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50444542-9319-4865-9ee1-b08ea60984af" (UID: "50444542-9319-4865-9ee1-b08ea60984af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:50:50.488001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.487935    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50444542-9319-4865-9ee1-b08ea60984af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:50:50.488001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.487957    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50444542-9319-4865-9ee1-b08ea60984af-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:50:50.488001 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.487967    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krk5g\" (UniqueName: \"kubernetes.io/projected/50444542-9319-4865-9ee1-b08ea60984af-kube-api-access-krk5g\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:50:50.938077 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938043    2576 generic.go:356] "Generic (PLEG): container finished" podID="50444542-9319-4865-9ee1-b08ea60984af" containerID="dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf" exitCode=2
Jul 28 12:50:50.938077 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938064    2576 generic.go:356] "Generic (PLEG): container finished" podID="50444542-9319-4865-9ee1-b08ea60984af" containerID="c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c" exitCode=2
Jul 28 12:50:50.938589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938098    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"
Jul 28 12:50:50.938589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" event={"ID":"50444542-9319-4865-9ee1-b08ea60984af","Type":"ContainerDied","Data":"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"}
Jul 28 12:50:50.938589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" event={"ID":"50444542-9319-4865-9ee1-b08ea60984af","Type":"ContainerDied","Data":"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"}
Jul 28 12:50:50.938589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb" event={"ID":"50444542-9319-4865-9ee1-b08ea60984af","Type":"ContainerDied","Data":"3f665d96193019c0c9dac0d7564dce0b13a652ea836518b4408d068c660a5bc4"}
Jul 28 12:50:50.938589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.938239    2576 scope.go:117] "RemoveContainer" containerID="dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"
Jul 28 12:50:50.940730 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.940709    2576 generic.go:356] "Generic (PLEG): container finished" podID="3bb8d456-34d6-4c1a-af28-76142161d439" containerID="7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6" exitCode=2
Jul 28 12:50:50.940851 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.940778    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerDied","Data":"7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6"}
Jul 28 12:50:50.942258 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.942235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerStarted","Data":"6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76"}
Jul 28 12:50:50.942383 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.942287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerStarted","Data":"f4ce669bd5c0b20fa5380baeeb9e9be56ab55c903173bbe2d8ec9572931e7d2a"}
Jul 28 12:50:50.949122 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.949105    2576 scope.go:117] "RemoveContainer" containerID="c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"
Jul 28 12:50:50.956080 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956064    2576 scope.go:117] "RemoveContainer" containerID="dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"
Jul 28 12:50:50.956308 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:50:50.956289    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf\": container with ID starting with dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf not found: ID does not exist" containerID="dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"
Jul 28 12:50:50.956415 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956319    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"} err="failed to get container status \"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf\": rpc error: code = NotFound desc = could not find container \"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf\": container with ID starting with dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf not found: ID does not exist"
Jul 28 12:50:50.956415 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956343    2576 scope.go:117] "RemoveContainer" containerID="c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"
Jul 28 12:50:50.956611 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:50:50.956595    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c\": container with ID starting with c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c not found: ID does not exist" containerID="c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"
Jul 28 12:50:50.956647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956617    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"} err="failed to get container status \"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c\": rpc error: code = NotFound desc = could not find container \"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c\": container with ID starting with c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c not found: ID does not exist"
Jul 28 12:50:50.956647 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956634    2576 scope.go:117] "RemoveContainer" containerID="dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"
Jul 28 12:50:50.956818 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956799    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf"} err="failed to get container status \"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf\": rpc error: code = NotFound desc = could not find container \"dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf\": container with ID starting with dba78bc2feb8c923477486b326e187f29c79e1e6efc0028d76ffaaf7f232f3bf not found: ID does not exist"
Jul 28 12:50:50.956881 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956820    2576 scope.go:117] "RemoveContainer" containerID="c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"
Jul 28 12:50:50.957000 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.956981    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c"} err="failed to get container status \"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c\": rpc error: code = NotFound desc = could not find container \"c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c\": container with ID starting with c4dca2a913c83ce963f0a8f1c767749567936feec3a55ee57d09e7b85c2d1b7c not found: ID does not exist"
Jul 28 12:50:50.974739 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.974718    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"]
Jul 28 12:50:50.977579 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:50.977548    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-vv8rb"]
Jul 28 12:50:51.719759 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:51.719729    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50444542-9319-4865-9ee1-b08ea60984af" path="/var/lib/kubelet/pods/50444542-9319-4865-9ee1-b08ea60984af/volumes"
Jul 28 12:50:53.513089 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:53.513047    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 12:50:53.959287 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:53.959256    2576 generic.go:356] "Generic (PLEG): container finished" podID="3bb8d456-34d6-4c1a-af28-76142161d439" containerID="e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905" exitCode=0
Jul 28 12:50:53.959432 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:53.959319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerDied","Data":"e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905"}
Jul 28 12:50:58.513592 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:58.513552    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 12:50:58.519112 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:58.519081    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:50:58.519425 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:50:58.519403    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:51:03.513604 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:03.513550    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 12:51:03.514025 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:03.513709    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:51:08.513465 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:08.513427    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 12:51:08.518759 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:08.518732    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:51:08.519071 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:08.519052    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:51:12.045321 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:12.045250    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerID="6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76" exitCode=0
Jul 28 12:51:12.045702 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:12.045329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerDied","Data":"6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76"}
Jul 28 12:51:12.046643 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:12.046629    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:51:13.513554 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:13.513269    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 12:51:18.513690 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:18.513650    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 12:51:18.518531 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:18.518492    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 12:51:18.518696 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:18.518634    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:51:18.518826 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:18.518796    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 12:51:18.518940 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:18.518925    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:51:20.062183 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.062152    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:51:20.096212 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.096185    2576 generic.go:356] "Generic (PLEG): container finished" podID="3bb8d456-34d6-4c1a-af28-76142161d439" containerID="a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58" exitCode=0
Jul 28 12:51:20.096405 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.096264    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"
Jul 28 12:51:20.096405 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.096268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerDied","Data":"a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58"}
Jul 28 12:51:20.096405 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.096396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt" event={"ID":"3bb8d456-34d6-4c1a-af28-76142161d439","Type":"ContainerDied","Data":"23c85ca05d0cbf7376011b8675ea212405b5b49177715c6bacecc9aeb2ee32ad"}
Jul 28 12:51:20.096586 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.096449    2576 scope.go:117] "RemoveContainer" containerID="a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58"
Jul 28 12:51:20.106336 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.106306    2576 scope.go:117] "RemoveContainer" containerID="7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6"
Jul 28 12:51:20.114888 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.114871    2576 scope.go:117] "RemoveContainer" containerID="e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905"
Jul 28 12:51:20.122645 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.122631    2576 scope.go:117] "RemoveContainer" containerID="e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996"
Jul 28 12:51:20.129854 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.129840    2576 scope.go:117] "RemoveContainer" containerID="a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58"
Jul 28 12:51:20.130186 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:51:20.130159    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58\": container with ID starting with a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58 not found: ID does not exist" containerID="a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58"
Jul 28 12:51:20.130264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.130192    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58"} err="failed to get container status \"a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58\": rpc error: code = NotFound desc = could not find container \"a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58\": container with ID starting with a4b3ce26d5a09c4b6e00b10061c34fc9add3a3655c0202d561873efaecb7ac58 not found: ID does not exist"
Jul 28 12:51:20.130264 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.130217    2576 scope.go:117] "RemoveContainer" containerID="7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6"
Jul 28 12:51:20.130480 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:51:20.130463    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6\": container with ID starting with 7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6 not found: ID does not exist" containerID="7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6"
Jul 28 12:51:20.130557 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.130486    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6"} err="failed to get container status \"7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6\": rpc error: code = NotFound desc = could not find container \"7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6\": container with ID starting with 7831fcfa30065cdaf714690c8136d7b4ff9f42b4a29c8238747f7455831f8ac6 not found: ID does not exist"
Jul 28 12:51:20.130557 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.130526    2576 scope.go:117] "RemoveContainer" containerID="e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905"
Jul 28 12:51:20.130931 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:51:20.130906    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905\": container with ID starting with e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905 not found: ID does not exist" containerID="e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905"
Jul 28 12:51:20.131009 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.130937    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905"} err="failed to get container status \"e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905\": rpc error: code = NotFound desc = could not find container \"e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905\": container with ID starting with e4a8d384fb283d07f064735740e42e804cabdace368009491bc88d039e199905 not found: ID does not exist"
Jul 28 12:51:20.131009 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.130964    2576 scope.go:117] "RemoveContainer" containerID="e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996"
Jul 28 12:51:20.131222 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:51:20.131202    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996\": container with ID starting with e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996 not found: ID does not exist" containerID="e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996"
Jul 28 12:51:20.131282 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.131230    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996"} err="failed to get container status \"e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996\": rpc error: code = NotFound desc = could not find container \"e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996\": container with ID starting with e0d8f11c463e3f7e61247aa27a1ccb84fdd7615f01a1b817940783eb93e3d996 not found: ID does not exist"
Jul 28 12:51:20.134552 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.134535    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls\") pod \"3bb8d456-34d6-4c1a-af28-76142161d439\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") "
Jul 28 12:51:20.134674 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.134590    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-kube-rbac-proxy-sar-config\") pod \"3bb8d456-34d6-4c1a-af28-76142161d439\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") "
Jul 28 12:51:20.134674 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.134620    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb9fz\" (UniqueName: \"kubernetes.io/projected/3bb8d456-34d6-4c1a-af28-76142161d439-kube-api-access-nb9fz\") pod \"3bb8d456-34d6-4c1a-af28-76142161d439\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") "
Jul 28 12:51:20.134794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.134686    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-cabundle-cert\") pod \"3bb8d456-34d6-4c1a-af28-76142161d439\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") "
Jul 28 12:51:20.134794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.134733    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb8d456-34d6-4c1a-af28-76142161d439-kserve-provision-location\") pod \"3bb8d456-34d6-4c1a-af28-76142161d439\" (UID: \"3bb8d456-34d6-4c1a-af28-76142161d439\") "
Jul 28 12:51:20.135036 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.135009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bb8d456-34d6-4c1a-af28-76142161d439" (UID: "3bb8d456-34d6-4c1a-af28-76142161d439"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:51:20.135202 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.135090    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3bb8d456-34d6-4c1a-af28-76142161d439" (UID: "3bb8d456-34d6-4c1a-af28-76142161d439"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:51:20.135293 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.135208    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb8d456-34d6-4c1a-af28-76142161d439-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bb8d456-34d6-4c1a-af28-76142161d439" (UID: "3bb8d456-34d6-4c1a-af28-76142161d439"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:51:20.136980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.136957    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bb8d456-34d6-4c1a-af28-76142161d439" (UID: "3bb8d456-34d6-4c1a-af28-76142161d439"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:51:20.137079 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.137013    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb8d456-34d6-4c1a-af28-76142161d439-kube-api-access-nb9fz" (OuterVolumeSpecName: "kube-api-access-nb9fz") pod "3bb8d456-34d6-4c1a-af28-76142161d439" (UID: "3bb8d456-34d6-4c1a-af28-76142161d439"). InnerVolumeSpecName "kube-api-access-nb9fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:51:20.236075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.235995    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:51:20.236075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.236029    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb8d456-34d6-4c1a-af28-76142161d439-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:51:20.236075 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.236070    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb8d456-34d6-4c1a-af28-76142161d439-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:51:20.236275 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.236087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb8d456-34d6-4c1a-af28-76142161d439-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:51:20.236275 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.236101    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb9fz\" (UniqueName: \"kubernetes.io/projected/3bb8d456-34d6-4c1a-af28-76142161d439-kube-api-access-nb9fz\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:51:20.423991 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.423957    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"]
Jul 28 12:51:20.429414 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:20.429364    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-65bf8d84bc-mvrkt"]
Jul 28 12:51:21.722263 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:51:21.722222    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" path="/var/lib/kubelet/pods/3bb8d456-34d6-4c1a-af28-76142161d439/volumes"
Jul 28 12:53:08.651492 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:08.651459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerStarted","Data":"bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5"}
Jul 28 12:53:09.658551 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:09.658516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerStarted","Data":"54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67"}
Jul 28 12:53:09.659050 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:09.658578    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:53:09.659050 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:09.658627    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:53:09.660279 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:09.660256    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:53:09.687836 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:09.687785    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" podStartSLOduration=2.865993439 podStartE2EDuration="2m20.687768923s" totalImagesPullingTime="1m56.501135541s" totalInitContainerRuntime="21.320639943s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:09.686852388 +0000 UTC m=+1082.587996778" watchObservedRunningTime="2026-07-28 12:53:09.687768923 +0000 UTC m=+1082.588913317"
Jul 28 12:53:10.662862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:10.662818    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:53:15.667394 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:15.667349    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:53:15.667980 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:15.667947    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 12:53:25.669290 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:25.669259    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:53:36.075997 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:36.075922    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"]
Jul 28 12:53:36.076346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:36.076156    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container" containerID="cri-o://bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5" gracePeriod=30
Jul 28 12:53:36.076346 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:36.076185    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kube-rbac-proxy" containerID="cri-o://54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67" gracePeriod=30
Jul 28 12:53:36.781169 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:36.781128    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerID="54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67" exitCode=2
Jul 28 12:53:36.781351 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:36.781203    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerDied","Data":"54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67"}
Jul 28 12:53:37.337602 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.337579    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:53:37.493889 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.493831    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ace1d49-beb1-45ef-80f6-0a4db1412621-kserve-provision-location\") pod \"6ace1d49-beb1-45ef-80f6-0a4db1412621\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") "
Jul 28 12:53:37.493889 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.493861    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-rbac-proxy-sar-config\") pod \"6ace1d49-beb1-45ef-80f6-0a4db1412621\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") "
Jul 28 12:53:37.494088 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.493930    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6lcn\" (UniqueName: \"kubernetes.io/projected/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-api-access-h6lcn\") pod \"6ace1d49-beb1-45ef-80f6-0a4db1412621\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") "
Jul 28 12:53:37.494088 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.493973    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ace1d49-beb1-45ef-80f6-0a4db1412621-proxy-tls\") pod \"6ace1d49-beb1-45ef-80f6-0a4db1412621\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") "
Jul 28 12:53:37.494088 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.493989    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-cabundle-cert\") pod \"6ace1d49-beb1-45ef-80f6-0a4db1412621\" (UID: \"6ace1d49-beb1-45ef-80f6-0a4db1412621\") "
Jul 28 12:53:37.494343 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.494313    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ace1d49-beb1-45ef-80f6-0a4db1412621" (UID: "6ace1d49-beb1-45ef-80f6-0a4db1412621"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:53:37.494460 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.494347    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ace1d49-beb1-45ef-80f6-0a4db1412621" (UID: "6ace1d49-beb1-45ef-80f6-0a4db1412621"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:53:37.496023 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.495992    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-api-access-h6lcn" (OuterVolumeSpecName: "kube-api-access-h6lcn") pod "6ace1d49-beb1-45ef-80f6-0a4db1412621" (UID: "6ace1d49-beb1-45ef-80f6-0a4db1412621"). InnerVolumeSpecName "kube-api-access-h6lcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:53:37.496023 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.496009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ace1d49-beb1-45ef-80f6-0a4db1412621-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ace1d49-beb1-45ef-80f6-0a4db1412621" (UID: "6ace1d49-beb1-45ef-80f6-0a4db1412621"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:53:37.498513 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.498492    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ace1d49-beb1-45ef-80f6-0a4db1412621-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ace1d49-beb1-45ef-80f6-0a4db1412621" (UID: "6ace1d49-beb1-45ef-80f6-0a4db1412621"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:53:37.595060 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.595032    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ace1d49-beb1-45ef-80f6-0a4db1412621-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:53:37.595060 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.595063    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:53:37.595234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.595078    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ace1d49-beb1-45ef-80f6-0a4db1412621-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:53:37.595234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.595092    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:53:37.595234 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.595105    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6lcn\" (UniqueName: \"kubernetes.io/projected/6ace1d49-beb1-45ef-80f6-0a4db1412621-kube-api-access-h6lcn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:53:37.786928 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.786851    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerID="bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5" exitCode=0
Jul 28 12:53:37.786928 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.786918    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"
Jul 28 12:53:37.787084 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.786917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerDied","Data":"bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5"}
Jul 28 12:53:37.787084 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.787040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz" event={"ID":"6ace1d49-beb1-45ef-80f6-0a4db1412621","Type":"ContainerDied","Data":"f4ce669bd5c0b20fa5380baeeb9e9be56ab55c903173bbe2d8ec9572931e7d2a"}
Jul 28 12:53:37.787084 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.787075    2576 scope.go:117] "RemoveContainer" containerID="54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67"
Jul 28 12:53:37.795225 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.795208    2576 scope.go:117] "RemoveContainer" containerID="bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5"
Jul 28 12:53:37.802448 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.802429    2576 scope.go:117] "RemoveContainer" containerID="6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76"
Jul 28 12:53:37.806453 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.806431    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"]
Jul 28 12:53:37.810386 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.810353    2576 scope.go:117] "RemoveContainer" containerID="54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67"
Jul 28 12:53:37.810650 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:53:37.810629    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67\": container with ID starting with 54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67 not found: ID does not exist" containerID="54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67"
Jul 28 12:53:37.810704 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.810658    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67"} err="failed to get container status \"54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67\": rpc error: code = NotFound desc = could not find container \"54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67\": container with ID starting with 54debb648abe72b1a20724593cee6b3445b4886b6511d889dba96646d5236d67 not found: ID does not exist"
Jul 28 12:53:37.810704 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.810675    2576 scope.go:117] "RemoveContainer" containerID="bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5"
Jul 28 12:53:37.810871 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.810853    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-lphsz"]
Jul 28 12:53:37.810922 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:53:37.810896    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5\": container with ID starting with bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5 not found: ID does not exist" containerID="bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5"
Jul 28 12:53:37.810922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.810910    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5"} err="failed to get container status \"bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5\": rpc error: code = NotFound desc = could not find container \"bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5\": container with ID starting with bf4efc2d3a9b5ee946da8a4ab32a389f6cf9c4118c7d778b1acd696cc715cec5 not found: ID does not exist"
Jul 28 12:53:37.810991 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.810922    2576 scope.go:117] "RemoveContainer" containerID="6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76"
Jul 28 12:53:37.811114 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:53:37.811096    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76\": container with ID starting with 6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76 not found: ID does not exist" containerID="6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76"
Jul 28 12:53:37.811152 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:37.811117    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76"} err="failed to get container status \"6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76\": rpc error: code = NotFound desc = could not find container \"6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76\": container with ID starting with 6faf5d6b0d4eaa94eff8b38ec84757312ba45beb0f876e7fb0e2b7acaf0b2b76 not found: ID does not exist"
Jul 28 12:53:39.719809 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:39.719779    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" path="/var/lib/kubelet/pods/6ace1d49-beb1-45ef-80f6-0a4db1412621/volumes"
Jul 28 12:53:41.275992 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.275960    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"]
Jul 28 12:53:41.276335 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276323    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent"
Jul 28 12:53:41.276335 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276334    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276348    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276354    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276361    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276380    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kserve-container"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276405    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kserve-container"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276411    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276416    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276424    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="storage-initializer"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276428    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="storage-initializer"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276436    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="storage-initializer"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276442    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="storage-initializer"
Jul 28 12:53:41.276447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276450    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276457    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276471    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276477    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276525    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kserve-container"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276532    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kserve-container"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276540    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276549    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50444542-9319-4865-9ee1-b08ea60984af" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276557    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ace1d49-beb1-45ef-80f6-0a4db1412621" containerName="kube-rbac-proxy"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276582    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="kserve-container"
Jul 28 12:53:41.276862 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.276595    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb8d456-34d6-4c1a-af28-76142161d439" containerName="agent"
Jul 28 12:53:41.305663 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.305643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"]
Jul 28 12:53:41.305794 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.305753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.309632 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.309612    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 12:53:41.310208 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.310184    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:53:41.310208 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.310206    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:53:41.310365 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.310242    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 12:53:41.310365 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.310196    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:53:41.311648 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.311627    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:53:41.311786 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.311683    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 12:53:41.418633 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.418609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.418741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.418637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvkg\" (UniqueName: \"kubernetes.io/projected/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-api-access-dzvkg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.418741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.418659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7a6848-3112-4ef9-b36b-4edad46d8fea-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.418741 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.418682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.418853 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.418760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520152 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvkg\" (UniqueName: \"kubernetes.io/projected/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-api-access-dzvkg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7a6848-3112-4ef9-b36b-4edad46d8fea-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520251 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520651 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520845 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.520895 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.520850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.522620 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.522604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7a6848-3112-4ef9-b36b-4edad46d8fea-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.529500 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.529444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvkg\" (UniqueName: \"kubernetes.io/projected/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-api-access-dzvkg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6tkvk\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.616262 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.616239    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:53:41.753274 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.753249    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"]
Jul 28 12:53:41.755586 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:53:41.755561    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7a6848_3112_4ef9_b36b_4edad46d8fea.slice/crio-c1aba5a86bd1c26a1d4947227149e2fb50225177591afef05684a8f730311aa6 WatchSource:0}: Error finding container c1aba5a86bd1c26a1d4947227149e2fb50225177591afef05684a8f730311aa6: Status 404 returned error can't find the container with id c1aba5a86bd1c26a1d4947227149e2fb50225177591afef05684a8f730311aa6
Jul 28 12:53:41.807885 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:41.807859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerStarted","Data":"c1aba5a86bd1c26a1d4947227149e2fb50225177591afef05684a8f730311aa6"}
Jul 28 12:53:42.814831 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:53:42.814789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerStarted","Data":"f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca"}
Jul 28 12:54:11.952733 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:11.952699    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerID="f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca" exitCode=0
Jul 28 12:54:11.952733 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:11.952736    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerDied","Data":"f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca"}
Jul 28 12:54:12.960720 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:12.960686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerStarted","Data":"d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642"}
Jul 28 12:54:12.961180 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:12.960748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerStarted","Data":"c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e"}
Jul 28 12:54:12.961180 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:12.960941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:54:12.986383 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:12.986327    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" podStartSLOduration=2.617900601 podStartE2EDuration="31.986310352s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.368409751s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:12.983536776 +0000 UTC m=+1145.884681179" watchObservedRunningTime="2026-07-28 12:54:12.986310352 +0000 UTC m=+1145.887454743"
Jul 28 12:54:13.966616 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:13.966587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:54:19.976937 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:54:19.976912    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:55:47.721315 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:47.721279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:55:57.324956 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:57.324925    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"]
Jul 28 12:55:57.325546 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:57.325161    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kserve-container" containerID="cri-o://c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e" gracePeriod=30
Jul 28 12:55:57.325546 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:57.325192    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kube-rbac-proxy" containerID="cri-o://d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642" gracePeriod=30
Jul 28 12:55:57.718120 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:57.718082    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.34:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 12:55:58.431283 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.431241    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerID="d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642" exitCode=2
Jul 28 12:55:58.431283 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.431263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerDied","Data":"d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642"}
Jul 28 12:55:58.567030 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.567009    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:55:58.626927 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.626874    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-cabundle-cert\") pod \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") "
Jul 28 12:55:58.626927 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.626907    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzvkg\" (UniqueName: \"kubernetes.io/projected/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-api-access-dzvkg\") pod \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") "
Jul 28 12:55:58.627121 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.626960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-rbac-proxy-sar-config\") pod \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") "
Jul 28 12:55:58.627121 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.626991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kserve-provision-location\") pod \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") "
Jul 28 12:55:58.627121 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.627025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7a6848-3112-4ef9-b36b-4edad46d8fea-proxy-tls\") pod \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\" (UID: \"0c7a6848-3112-4ef9-b36b-4edad46d8fea\") "
Jul 28 12:55:58.627298 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.627279    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c7a6848-3112-4ef9-b36b-4edad46d8fea" (UID: "0c7a6848-3112-4ef9-b36b-4edad46d8fea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:58.627351 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.627309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c7a6848-3112-4ef9-b36b-4edad46d8fea" (UID: "0c7a6848-3112-4ef9-b36b-4edad46d8fea"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:55:58.628981 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.628958    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-api-access-dzvkg" (OuterVolumeSpecName: "kube-api-access-dzvkg") pod "0c7a6848-3112-4ef9-b36b-4edad46d8fea" (UID: "0c7a6848-3112-4ef9-b36b-4edad46d8fea"). InnerVolumeSpecName "kube-api-access-dzvkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:55:58.629078 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.628987    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7a6848-3112-4ef9-b36b-4edad46d8fea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c7a6848-3112-4ef9-b36b-4edad46d8fea" (UID: "0c7a6848-3112-4ef9-b36b-4edad46d8fea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:55:58.631983 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.631963    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c7a6848-3112-4ef9-b36b-4edad46d8fea" (UID: "0c7a6848-3112-4ef9-b36b-4edad46d8fea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:55:58.728513 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.728492    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:55:58.728513 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.728512    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvkg\" (UniqueName: \"kubernetes.io/projected/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-api-access-dzvkg\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:55:58.728651 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.728522    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:55:58.728651 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.728531    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c7a6848-3112-4ef9-b36b-4edad46d8fea-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:55:58.728651 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:58.728540    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c7a6848-3112-4ef9-b36b-4edad46d8fea-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:55:59.437859 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.437826    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerID="c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e" exitCode=0
Jul 28 12:55:59.438294 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.437880    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"
Jul 28 12:55:59.438294 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.437895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerDied","Data":"c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e"}
Jul 28 12:55:59.438294 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.437945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk" event={"ID":"0c7a6848-3112-4ef9-b36b-4edad46d8fea","Type":"ContainerDied","Data":"c1aba5a86bd1c26a1d4947227149e2fb50225177591afef05684a8f730311aa6"}
Jul 28 12:55:59.438294 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.437970    2576 scope.go:117] "RemoveContainer" containerID="d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642"
Jul 28 12:55:59.447254 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.447095    2576 scope.go:117] "RemoveContainer" containerID="c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e"
Jul 28 12:55:59.454589 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.454574    2576 scope.go:117] "RemoveContainer" containerID="f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca"
Jul 28 12:55:59.461281 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.461260    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"]
Jul 28 12:55:59.462130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.462108    2576 scope.go:117] "RemoveContainer" containerID="d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642"
Jul 28 12:55:59.462380 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:55:59.462351    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642\": container with ID starting with d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642 not found: ID does not exist" containerID="d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642"
Jul 28 12:55:59.462432 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.462389    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642"} err="failed to get container status \"d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642\": rpc error: code = NotFound desc = could not find container \"d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642\": container with ID starting with d475455abb46a2f204cf5a035afafff07523aa8405aecc1053441a42828b1642 not found: ID does not exist"
Jul 28 12:55:59.462432 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.462404    2576 scope.go:117] "RemoveContainer" containerID="c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e"
Jul 28 12:55:59.462601 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:55:59.462582    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e\": container with ID starting with c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e not found: ID does not exist" containerID="c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e"
Jul 28 12:55:59.462676 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.462609    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e"} err="failed to get container status \"c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e\": rpc error: code = NotFound desc = could not find container \"c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e\": container with ID starting with c3749128bcce7296d06b825198b78c249121a3ca146a3eca625be228108cca1e not found: ID does not exist"
Jul 28 12:55:59.462676 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.462624    2576 scope.go:117] "RemoveContainer" containerID="f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca"
Jul 28 12:55:59.462901 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:55:59.462876    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca\": container with ID starting with f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca not found: ID does not exist" containerID="f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca"
Jul 28 12:55:59.463024 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.462910    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca"} err="failed to get container status \"f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca\": rpc error: code = NotFound desc = could not find container \"f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca\": container with ID starting with f0dde1dd24ca87a20f103fa7772569fc37276c391033864b53add5bcac86ddca not found: ID does not exist"
Jul 28 12:55:59.465082 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.465064    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6tkvk"]
Jul 28 12:55:59.720330 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:55:59.720269    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" path="/var/lib/kubelet/pods/0c7a6848-3112-4ef9-b36b-4edad46d8fea/volumes"
Jul 28 12:56:02.495341 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495312    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"]
Jul 28 12:56:02.495758 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495677    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kserve-container"
Jul 28 12:56:02.495758 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495693    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kserve-container"
Jul 28 12:56:02.495758 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495704    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="storage-initializer"
Jul 28 12:56:02.495758 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495709    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="storage-initializer"
Jul 28 12:56:02.495758 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495716    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kube-rbac-proxy"
Jul 28 12:56:02.495758 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495720    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kube-rbac-proxy"
Jul 28 12:56:02.495970 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495804    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kserve-container"
Jul 28 12:56:02.495970 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.495814    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c7a6848-3112-4ef9-b36b-4edad46d8fea" containerName="kube-rbac-proxy"
Jul 28 12:56:02.498840 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.498824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.501494 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501463    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:56:02.501628 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501615    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 12:56:02.501788 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501776    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 12:56:02.501876 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501859    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:56:02.501922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501887    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 12:56:02.501922 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:56:02.502013 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.501935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:56:02.510828 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.510807    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"]
Jul 28 12:56:02.554211 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.554193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.554325 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.554222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f555450-e6b3-443e-95cb-83abfe85f31d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.554325 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.554298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.554447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.554330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvplx\" (UniqueName: \"kubernetes.io/projected/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-api-access-bvplx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.554447 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.554365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f555450-e6b3-443e-95cb-83abfe85f31d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.655665 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.655645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.655784 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.655672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvplx\" (UniqueName: \"kubernetes.io/projected/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-api-access-bvplx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.655784 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.655693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f555450-e6b3-443e-95cb-83abfe85f31d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.655784 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.655729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.655784 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.655766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f555450-e6b3-443e-95cb-83abfe85f31d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.656149 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.656129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f555450-e6b3-443e-95cb-83abfe85f31d-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.656310 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.656286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.656479 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.656357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.658117 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.658102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f555450-e6b3-443e-95cb-83abfe85f31d-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.665658 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.665639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvplx\" (UniqueName: \"kubernetes.io/projected/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-api-access-bvplx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.809363 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.809307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:02.933351 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:02.933326    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"]
Jul 28 12:56:02.935732 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:56:02.935704    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f555450_e6b3_443e_95cb_83abfe85f31d.slice/crio-d66ac3b19014825670c31c9db9a56bc3be50dd99ae1ebccc0dc5c8f857657276 WatchSource:0}: Error finding container d66ac3b19014825670c31c9db9a56bc3be50dd99ae1ebccc0dc5c8f857657276: Status 404 returned error can't find the container with id d66ac3b19014825670c31c9db9a56bc3be50dd99ae1ebccc0dc5c8f857657276
Jul 28 12:56:03.457189 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:03.457156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerStarted","Data":"c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3"}
Jul 28 12:56:03.457356 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:03.457224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerStarted","Data":"d66ac3b19014825670c31c9db9a56bc3be50dd99ae1ebccc0dc5c8f857657276"}
Jul 28 12:56:08.483426 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:08.483389    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerID="c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3" exitCode=0
Jul 28 12:56:08.483795 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:08.483440    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerDied","Data":"c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3"}
Jul 28 12:56:09.490230 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:09.490189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerStarted","Data":"b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9"}
Jul 28 12:56:09.490623 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:09.490271    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerStarted","Data":"6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a"}
Jul 28 12:56:09.490623 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:09.490515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:09.513445 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:09.513397    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" podStartSLOduration=2.61891054 podStartE2EDuration="7.513364981s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.894454441s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:56:09.511016957 +0000 UTC m=+1262.412161348" watchObservedRunningTime="2026-07-28 12:56:09.513364981 +0000 UTC m=+1262.414509373"
Jul 28 12:56:10.494772 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:10.494743    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:56:16.503064 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:56:16.503035    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:57:44.720797 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:44.720710    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:57:55.756517 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:55.756479    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 12:57:59.047745 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:59.047704    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"]
Jul 28 12:57:59.048194 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:59.047958    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kserve-container" containerID="cri-o://6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a" gracePeriod=30
Jul 28 12:57:59.048194 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:59.048000    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kube-rbac-proxy" containerID="cri-o://b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9" gracePeriod=30
Jul 28 12:57:59.974676 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:59.974643    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerID="b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9" exitCode=2
Jul 28 12:57:59.974848 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:57:59.974732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerDied","Data":"b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9"}
Jul 28 12:58:00.290933 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.290910    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:58:00.426886 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.426865    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-rbac-proxy-sar-config\") pod \"4f555450-e6b3-443e-95cb-83abfe85f31d\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") "
Jul 28 12:58:00.427017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.426925    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f555450-e6b3-443e-95cb-83abfe85f31d-proxy-tls\") pod \"4f555450-e6b3-443e-95cb-83abfe85f31d\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") "
Jul 28 12:58:00.427017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.426952    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-cabundle-cert\") pod \"4f555450-e6b3-443e-95cb-83abfe85f31d\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") "
Jul 28 12:58:00.427017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.426982    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f555450-e6b3-443e-95cb-83abfe85f31d-kserve-provision-location\") pod \"4f555450-e6b3-443e-95cb-83abfe85f31d\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") "
Jul 28 12:58:00.427017 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.427004    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvplx\" (UniqueName: \"kubernetes.io/projected/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-api-access-bvplx\") pod \"4f555450-e6b3-443e-95cb-83abfe85f31d\" (UID: \"4f555450-e6b3-443e-95cb-83abfe85f31d\") "
Jul 28 12:58:00.427282 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.427257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f555450-e6b3-443e-95cb-83abfe85f31d" (UID: "4f555450-e6b3-443e-95cb-83abfe85f31d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:00.427408 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.427389    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f555450-e6b3-443e-95cb-83abfe85f31d" (UID: "4f555450-e6b3-443e-95cb-83abfe85f31d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:58:00.429042 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.429018    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f555450-e6b3-443e-95cb-83abfe85f31d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f555450-e6b3-443e-95cb-83abfe85f31d" (UID: "4f555450-e6b3-443e-95cb-83abfe85f31d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:58:00.429147 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.429135    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-api-access-bvplx" (OuterVolumeSpecName: "kube-api-access-bvplx") pod "4f555450-e6b3-443e-95cb-83abfe85f31d" (UID: "4f555450-e6b3-443e-95cb-83abfe85f31d"). InnerVolumeSpecName "kube-api-access-bvplx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:58:00.431795 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.431766    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f555450-e6b3-443e-95cb-83abfe85f31d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f555450-e6b3-443e-95cb-83abfe85f31d" (UID: "4f555450-e6b3-443e-95cb-83abfe85f31d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 12:58:00.527928 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.527870    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f555450-e6b3-443e-95cb-83abfe85f31d-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:58:00.527928 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.527892    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:58:00.527928 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.527905    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f555450-e6b3-443e-95cb-83abfe85f31d-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:58:00.527928 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.527920    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvplx\" (UniqueName: \"kubernetes.io/projected/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-api-access-bvplx\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:58:00.528111 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.527934    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f555450-e6b3-443e-95cb-83abfe85f31d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 12:58:00.981444 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.981420    2576 generic.go:356] "Generic (PLEG): container finished" podID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerID="6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a" exitCode=0
Jul 28 12:58:00.981565 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.981480    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"
Jul 28 12:58:00.981565 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.981507    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerDied","Data":"6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a"}
Jul 28 12:58:00.981641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.981572    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn" event={"ID":"4f555450-e6b3-443e-95cb-83abfe85f31d","Type":"ContainerDied","Data":"d66ac3b19014825670c31c9db9a56bc3be50dd99ae1ebccc0dc5c8f857657276"}
Jul 28 12:58:00.981641 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.981602    2576 scope.go:117] "RemoveContainer" containerID="b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9"
Jul 28 12:58:00.990764 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.990748    2576 scope.go:117] "RemoveContainer" containerID="6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a"
Jul 28 12:58:00.998773 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:00.998755    2576 scope.go:117] "RemoveContainer" containerID="c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3"
Jul 28 12:58:01.006104 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.006080    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"]
Jul 28 12:58:01.006840 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.006823    2576 scope.go:117] "RemoveContainer" containerID="b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9"
Jul 28 12:58:01.007143 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:58:01.007124    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9\": container with ID starting with b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9 not found: ID does not exist" containerID="b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9"
Jul 28 12:58:01.007203 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.007151    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9"} err="failed to get container status \"b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9\": rpc error: code = NotFound desc = could not find container \"b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9\": container with ID starting with b9d1681348255efcf528267aaa790f79e3de6bdee71dc2450a8b052f56cd2ac9 not found: ID does not exist"
Jul 28 12:58:01.007203 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.007168    2576 scope.go:117] "RemoveContainer" containerID="6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a"
Jul 28 12:58:01.007436 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:58:01.007420    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a\": container with ID starting with 6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a not found: ID does not exist" containerID="6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a"
Jul 28 12:58:01.007481 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.007442    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a"} err="failed to get container status \"6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a\": rpc error: code = NotFound desc = could not find container \"6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a\": container with ID starting with 6a852fc8d7eb022b9a25480ca7847754bbfa290e38e4a3ba6efcb67330ee9b1a not found: ID does not exist"
Jul 28 12:58:01.007481 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.007457    2576 scope.go:117] "RemoveContainer" containerID="c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3"
Jul 28 12:58:01.007679 ip-10-0-129-190 kubenswrapper[2576]: E0728 12:58:01.007665    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3\": container with ID starting with c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3 not found: ID does not exist" containerID="c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3"
Jul 28 12:58:01.007716 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.007682    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3"} err="failed to get container status \"c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3\": rpc error: code = NotFound desc = could not find container \"c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3\": container with ID starting with c07725ce35c0f8ecd29a27feb1ee6a13dd286678eda1614ef8cbd64e5071aae3 not found: ID does not exist"
Jul 28 12:58:01.007796 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.007779    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9vfpn"]
Jul 28 12:58:01.719780 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:01.719750    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" path="/var/lib/kubelet/pods/4f555450-e6b3-443e-95cb-83abfe85f31d/volumes"
Jul 28 12:58:04.186598 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186569    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"]
Jul 28 12:58:04.186945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186887    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kube-rbac-proxy"
Jul 28 12:58:04.186945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186898    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kube-rbac-proxy"
Jul 28 12:58:04.186945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186909    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="storage-initializer"
Jul 28 12:58:04.186945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186915    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="storage-initializer"
Jul 28 12:58:04.186945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186927    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kserve-container"
Jul 28 12:58:04.186945 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186934    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kserve-container"
Jul 28 12:58:04.187134 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186982    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kserve-container"
Jul 28 12:58:04.187134 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.186991    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f555450-e6b3-443e-95cb-83abfe85f31d" containerName="kube-rbac-proxy"
Jul 28 12:58:04.191895 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.191873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.194982 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.194960    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 12:58:04.195101 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.195012    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 12:58:04.195101 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.195026    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 12:58:04.195101 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.195026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 12:58:04.195101 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.194968    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 12:58:04.195503 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.195487    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 12:58:04.195598 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.195513    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 12:58:04.200833 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.200813    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"]
Jul 28 12:58:04.359515 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.359496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.359622 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.359527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f7cd30-f529-43d4-9058-0cf8c303320c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.359622 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.359553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.359696 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.359630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f7cd30-f529-43d4-9058-0cf8c303320c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.359696 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.359663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7cm\" (UniqueName: \"kubernetes.io/projected/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-api-access-tz7cm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.460361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.460298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f7cd30-f529-43d4-9058-0cf8c303320c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.460361 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.460332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7cm\" (UniqueName: \"kubernetes.io/projected/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-api-access-tz7cm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.460548 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.460404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.460548 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.460440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f7cd30-f529-43d4-9058-0cf8c303320c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.460548 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.460479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.460844 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.460824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f7cd30-f529-43d4-9058-0cf8c303320c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.461025 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.461004    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.461130 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.461112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.462675 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.462650    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f7cd30-f529-43d4-9058-0cf8c303320c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.470045 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.470025    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7cm\" (UniqueName: \"kubernetes.io/projected/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-api-access-tz7cm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.502947 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.502925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:04.637682 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:04.637658    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"]
Jul 28 12:58:04.640000 ip-10-0-129-190 kubenswrapper[2576]: W0728 12:58:04.639976    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f7cd30_f529_43d4_9058_0cf8c303320c.slice/crio-2b2b77cd74db133e0d734e55f9f99db0d9a1a7eb4405d8d126187888ee5755bc WatchSource:0}: Error finding container 2b2b77cd74db133e0d734e55f9f99db0d9a1a7eb4405d8d126187888ee5755bc: Status 404 returned error can't find the container with id 2b2b77cd74db133e0d734e55f9f99db0d9a1a7eb4405d8d126187888ee5755bc
Jul 28 12:58:05.000609 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:05.000567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerStarted","Data":"749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe"}
Jul 28 12:58:05.000800 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:05.000641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerStarted","Data":"2b2b77cd74db133e0d734e55f9f99db0d9a1a7eb4405d8d126187888ee5755bc"}
Jul 28 12:58:35.131420 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:35.131385    2576 generic.go:356] "Generic (PLEG): container finished" podID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerID="749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe" exitCode=0
Jul 28 12:58:35.131420 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:35.131397    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerDied","Data":"749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe"}
Jul 28 12:58:36.138364 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:36.138334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerStarted","Data":"6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a"}
Jul 28 12:58:36.138729 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:36.138416    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerStarted","Data":"bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515"}
Jul 28 12:58:36.138729 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:36.138597    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:36.160853 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:36.160806    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" podStartSLOduration=2.673023624 podStartE2EDuration="32.160794485s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.487770861s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:58:36.160207365 +0000 UTC m=+1409.061351754" watchObservedRunningTime="2026-07-28 12:58:36.160794485 +0000 UTC m=+1409.061938874"
Jul 28 12:58:37.143262 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:37.143227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 12:58:43.152231 ip-10-0-129-190 kubenswrapper[2576]: I0728 12:58:43.152206    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 13:00:09.720798 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:09.720771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 13:00:20.276292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:20.276249    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"]
Jul 28 13:00:20.276843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:20.276538    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kserve-container" containerID="cri-o://bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515" gracePeriod=30
Jul 28 13:00:20.276843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:20.276583    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kube-rbac-proxy" containerID="cri-o://6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a" gracePeriod=30
Jul 28 13:00:20.589248 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:20.589214    2576 generic.go:356] "Generic (PLEG): container finished" podID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerID="6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a" exitCode=2
Jul 28 13:00:20.589426 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:20.589295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerDied","Data":"6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a"}
Jul 28 13:00:21.527401 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.527362    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 13:00:21.595312 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.595284    2576 generic.go:356] "Generic (PLEG): container finished" podID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerID="bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515" exitCode=0
Jul 28 13:00:21.595455 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.595349    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"
Jul 28 13:00:21.595455 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.595358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerDied","Data":"bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515"}
Jul 28 13:00:21.595455 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.595429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49" event={"ID":"22f7cd30-f529-43d4-9058-0cf8c303320c","Type":"ContainerDied","Data":"2b2b77cd74db133e0d734e55f9f99db0d9a1a7eb4405d8d126187888ee5755bc"}
Jul 28 13:00:21.595620 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.595459    2576 scope.go:117] "RemoveContainer" containerID="6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a"
Jul 28 13:00:21.603356 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.603343    2576 scope.go:117] "RemoveContainer" containerID="bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515"
Jul 28 13:00:21.607513 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607493    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-cabundle-cert\") pod \"22f7cd30-f529-43d4-9058-0cf8c303320c\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") "
Jul 28 13:00:21.607604 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607523    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-rbac-proxy-sar-config\") pod \"22f7cd30-f529-43d4-9058-0cf8c303320c\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") "
Jul 28 13:00:21.607683 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607664    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz7cm\" (UniqueName: \"kubernetes.io/projected/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-api-access-tz7cm\") pod \"22f7cd30-f529-43d4-9058-0cf8c303320c\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") "
Jul 28 13:00:21.607752 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607719    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f7cd30-f529-43d4-9058-0cf8c303320c-proxy-tls\") pod \"22f7cd30-f529-43d4-9058-0cf8c303320c\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") "
Jul 28 13:00:21.607805 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607778    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f7cd30-f529-43d4-9058-0cf8c303320c-kserve-provision-location\") pod \"22f7cd30-f529-43d4-9058-0cf8c303320c\" (UID: \"22f7cd30-f529-43d4-9058-0cf8c303320c\") "
Jul 28 13:00:21.607883 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607863    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22f7cd30-f529-43d4-9058-0cf8c303320c" (UID: "22f7cd30-f529-43d4-9058-0cf8c303320c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:00:21.607938 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.607869    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22f7cd30-f529-43d4-9058-0cf8c303320c" (UID: "22f7cd30-f529-43d4-9058-0cf8c303320c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:00:21.608053 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.608037    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:00:21.608105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.608058    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:00:21.609747 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.609726    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-api-access-tz7cm" (OuterVolumeSpecName: "kube-api-access-tz7cm") pod "22f7cd30-f529-43d4-9058-0cf8c303320c" (UID: "22f7cd30-f529-43d4-9058-0cf8c303320c"). InnerVolumeSpecName "kube-api-access-tz7cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:00:21.610059 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.610035    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f7cd30-f529-43d4-9058-0cf8c303320c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22f7cd30-f529-43d4-9058-0cf8c303320c" (UID: "22f7cd30-f529-43d4-9058-0cf8c303320c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:00:21.611184 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.611164    2576 scope.go:117] "RemoveContainer" containerID="749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe"
Jul 28 13:00:21.612892 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.612870    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f7cd30-f529-43d4-9058-0cf8c303320c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22f7cd30-f529-43d4-9058-0cf8c303320c" (UID: "22f7cd30-f529-43d4-9058-0cf8c303320c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:00:21.623806 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.623787    2576 scope.go:117] "RemoveContainer" containerID="6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a"
Jul 28 13:00:21.624036 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:00:21.624017    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a\": container with ID starting with 6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a not found: ID does not exist" containerID="6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a"
Jul 28 13:00:21.624083 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.624045    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a"} err="failed to get container status \"6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a\": rpc error: code = NotFound desc = could not find container \"6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a\": container with ID starting with 6d63a742acf024c68ebf1b0bd613016b139c0912415b5ee22a570d626a61f50a not found: ID does not exist"
Jul 28 13:00:21.624083 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.624062    2576 scope.go:117] "RemoveContainer" containerID="bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515"
Jul 28 13:00:21.624323 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:00:21.624306    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515\": container with ID starting with bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515 not found: ID does not exist" containerID="bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515"
Jul 28 13:00:21.624411 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.624332    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515"} err="failed to get container status \"bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515\": rpc error: code = NotFound desc = could not find container \"bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515\": container with ID starting with bf8aef0047ad55999ac0766eafd9b49d648135669e5ace526c142afe12948515 not found: ID does not exist"
Jul 28 13:00:21.624411 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.624354    2576 scope.go:117] "RemoveContainer" containerID="749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe"
Jul 28 13:00:21.624613 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:00:21.624596    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe\": container with ID starting with 749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe not found: ID does not exist" containerID="749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe"
Jul 28 13:00:21.624687 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.624618    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe"} err="failed to get container status \"749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe\": rpc error: code = NotFound desc = could not find container \"749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe\": container with ID starting with 749af5080b15f2e1c5fedece75668e83b1503ac8d7fd11eb1f937bbd3ff9fabe not found: ID does not exist"
Jul 28 13:00:21.709157 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.709137    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7cm\" (UniqueName: \"kubernetes.io/projected/22f7cd30-f529-43d4-9058-0cf8c303320c-kube-api-access-tz7cm\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:00:21.709157 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.709156    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f7cd30-f529-43d4-9058-0cf8c303320c-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:00:21.709268 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.709165    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22f7cd30-f529-43d4-9058-0cf8c303320c-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:00:21.912023 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.911961    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"]
Jul 28 13:00:21.915684 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:21.915664    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-xjn49"]
Jul 28 13:00:23.720892 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:23.720855    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" path="/var/lib/kubelet/pods/22f7cd30-f529-43d4-9058-0cf8c303320c/volumes"
Jul 28 13:00:25.484439 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484405    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"]
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kserve-container"
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484765    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kserve-container"
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kube-rbac-proxy"
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484780    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kube-rbac-proxy"
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484798    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="storage-initializer"
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484808    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="storage-initializer"
Jul 28 13:00:25.484860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484862    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kube-rbac-proxy"
Jul 28 13:00:25.485085 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.484871    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f7cd30-f529-43d4-9058-0cf8c303320c" containerName="kserve-container"
Jul 28 13:00:25.489573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.489553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.492996 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.492978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:00:25.493760 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.493743    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 13:00:25.493855 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.493787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:00:25.494319 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.494300    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:00:25.494387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.494308    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:00:25.494387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.494308    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 13:00:25.494387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.494353    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:00:25.498968 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.498945    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"]
Jul 28 13:00:25.638938 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.638909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.639046 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.638986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.639046 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.639021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtdn\" (UniqueName: \"kubernetes.io/projected/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-api-access-gbtdn\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.639140 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.639060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.639140 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.639088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.739764 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.739703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.739764 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.739743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtdn\" (UniqueName: \"kubernetes.io/projected/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-api-access-gbtdn\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.739932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.739789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.739932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.739822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.739932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.739891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.740258 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.740235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.740559 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.740542    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.740640 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.740568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.742190 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.742173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.748501 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.748480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtdn\" (UniqueName: \"kubernetes.io/projected/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-api-access-gbtdn\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.801587 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.801564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:25.930592 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:25.930568    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"]
Jul 28 13:00:26.622005 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:26.621973    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerStarted","Data":"c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f"}
Jul 28 13:00:26.622422 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:26.622034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerStarted","Data":"b6da9502a128982db3e18e5e63f2bd4e757f7b8eee2f631cfbbd07b1d0aef61f"}
Jul 28 13:00:34.658874 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:34.658844    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerID="c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f" exitCode=0
Jul 28 13:00:34.659233 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:34.658886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerDied","Data":"c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f"}
Jul 28 13:00:35.665822 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:35.665784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerStarted","Data":"c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a"}
Jul 28 13:00:35.666222 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:35.665871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerStarted","Data":"1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf"}
Jul 28 13:00:35.666222 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:35.666148    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:35.666351 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:35.666313    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:35.667823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:35.667794    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:00:35.687180 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:35.687138    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podStartSLOduration=2.999056753 podStartE2EDuration="10.6871256s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.688068847s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:35.68499353 +0000 UTC m=+1528.586137920" watchObservedRunningTime="2026-07-28 13:00:35.6871256 +0000 UTC m=+1528.588269990"
Jul 28 13:00:36.671618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:36.671576    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:00:37.675835 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:37.675788    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:00:42.681590 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:42.681561    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:00:42.682094 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:42.682067    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:00:52.683198 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:00:52.683161    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:01:08.715945 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:08.715905    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"]
Jul 28 13:01:08.716525 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:08.716241    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container" containerID="cri-o://1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf" gracePeriod=30
Jul 28 13:01:08.716525 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:08.716473    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kube-rbac-proxy" containerID="cri-o://c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a" gracePeriod=30
Jul 28 13:01:09.832354 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:09.832322    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerID="c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a" exitCode=2
Jul 28 13:01:09.832758 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:09.832399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerDied","Data":"c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a"}
Jul 28 13:01:10.765283 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.765258    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:01:10.838971 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.838938    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerID="1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf" exitCode=0
Jul 28 13:01:10.839353 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.839000    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"
Jul 28 13:01:10.839353 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.838993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerDied","Data":"1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf"}
Jul 28 13:01:10.839353 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.839140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b" event={"ID":"e8b97b2c-557d-4cb5-92d1-d1c496d42d86","Type":"ContainerDied","Data":"b6da9502a128982db3e18e5e63f2bd4e757f7b8eee2f631cfbbd07b1d0aef61f"}
Jul 28 13:01:10.839353 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.839182    2576 scope.go:117] "RemoveContainer" containerID="c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a"
Jul 28 13:01:10.847602 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.847581    2576 scope.go:117] "RemoveContainer" containerID="1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf"
Jul 28 13:01:10.855429 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.855406    2576 scope.go:117] "RemoveContainer" containerID="c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f"
Jul 28 13:01:10.863446 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.863427    2576 scope.go:117] "RemoveContainer" containerID="c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a"
Jul 28 13:01:10.863783 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:10.863763    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a\": container with ID starting with c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a not found: ID does not exist" containerID="c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a"
Jul 28 13:01:10.863865 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.863799    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a"} err="failed to get container status \"c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a\": rpc error: code = NotFound desc = could not find container \"c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a\": container with ID starting with c9569bf698f583123714cb618366eb76f6def21b82d7d5ecf4f0515c3795a30a not found: ID does not exist"
Jul 28 13:01:10.863865 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.863827    2576 scope.go:117] "RemoveContainer" containerID="1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf"
Jul 28 13:01:10.864074 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:10.864057    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf\": container with ID starting with 1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf not found: ID does not exist" containerID="1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf"
Jul 28 13:01:10.864158 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.864080    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf"} err="failed to get container status \"1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf\": rpc error: code = NotFound desc = could not find container \"1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf\": container with ID starting with 1b5eaf6458db904809d90e37778f0ed07ab5c35e97ef0af0f7ff0edad7da5baf not found: ID does not exist"
Jul 28 13:01:10.864158 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.864096    2576 scope.go:117] "RemoveContainer" containerID="c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f"
Jul 28 13:01:10.864312 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:10.864295    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f\": container with ID starting with c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f not found: ID does not exist" containerID="c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f"
Jul 28 13:01:10.864356 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.864316    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f"} err="failed to get container status \"c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f\": rpc error: code = NotFound desc = could not find container \"c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f\": container with ID starting with c18c653e348ccd8e74a8088dc57f30e1af234f23c94415a756e89628f7f1b95f not found: ID does not exist"
Jul 28 13:01:10.926303 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926264    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbtdn\" (UniqueName: \"kubernetes.io/projected/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-api-access-gbtdn\") pod \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") "
Jul 28 13:01:10.926504 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926332    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-cabundle-cert\") pod \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") "
Jul 28 13:01:10.926504 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-proxy-tls\") pod \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") "
Jul 28 13:01:10.926504 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926421    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kserve-provision-location\") pod \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") "
Jul 28 13:01:10.926504 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926477    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-rbac-proxy-sar-config\") pod \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\" (UID: \"e8b97b2c-557d-4cb5-92d1-d1c496d42d86\") "
Jul 28 13:01:10.926817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926789    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8b97b2c-557d-4cb5-92d1-d1c496d42d86" (UID: "e8b97b2c-557d-4cb5-92d1-d1c496d42d86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:10.926933 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.926879    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8b97b2c-557d-4cb5-92d1-d1c496d42d86" (UID: "e8b97b2c-557d-4cb5-92d1-d1c496d42d86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:10.927355 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.927334    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8b97b2c-557d-4cb5-92d1-d1c496d42d86" (UID: "e8b97b2c-557d-4cb5-92d1-d1c496d42d86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:01:10.928560 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.928532    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8b97b2c-557d-4cb5-92d1-d1c496d42d86" (UID: "e8b97b2c-557d-4cb5-92d1-d1c496d42d86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:10.928560 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:10.928538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-api-access-gbtdn" (OuterVolumeSpecName: "kube-api-access-gbtdn") pod "e8b97b2c-557d-4cb5-92d1-d1c496d42d86" (UID: "e8b97b2c-557d-4cb5-92d1-d1c496d42d86"). InnerVolumeSpecName "kube-api-access-gbtdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:01:11.027639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.027604    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:11.027639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.027635    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtdn\" (UniqueName: \"kubernetes.io/projected/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kube-api-access-gbtdn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:11.027639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.027646    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:11.027894 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.027656    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:11.027894 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.027664    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8b97b2c-557d-4cb5-92d1-d1c496d42d86-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:11.163739 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.163699    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"]
Jul 28 13:01:11.166102 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.166075    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-45v7b"]
Jul 28 13:01:11.720062 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:11.720026    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" path="/var/lib/kubelet/pods/e8b97b2c-557d-4cb5-92d1-d1c496d42d86/volumes"
Jul 28 13:01:13.906714 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.906668    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"]
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907023    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kube-rbac-proxy"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907034    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kube-rbac-proxy"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907051    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="storage-initializer"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907057    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="storage-initializer"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907064    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907071    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907124    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kserve-container"
Jul 28 13:01:13.907172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.907138    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b97b2c-557d-4cb5-92d1-d1c496d42d86" containerName="kube-rbac-proxy"
Jul 28 13:01:13.911092 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.911072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:13.913713 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.913686    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:01:13.913844 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.913768    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 13:01:13.913844 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.913796    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:01:13.914161 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.913898    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:01:13.914161 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.914143    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 13:01:13.914269 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.914144    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:01:13.914962 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.914943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:01:13.921692 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:13.921665    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"]
Jul 28 13:01:14.055494 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.055451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.055684 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.055512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c3713f4-590a-4690-9c0a-68827221721a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.055684 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.055587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhn4\" (UniqueName: \"kubernetes.io/projected/4c3713f4-590a-4690-9c0a-68827221721a-kube-api-access-lwhn4\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.055684 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.055669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.055829 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.055712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c3713f4-590a-4690-9c0a-68827221721a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.156853 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.156757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.156853 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.156800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c3713f4-590a-4690-9c0a-68827221721a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.156853 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.156840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.157118 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.156877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c3713f4-590a-4690-9c0a-68827221721a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.157118 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.156898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhn4\" (UniqueName: \"kubernetes.io/projected/4c3713f4-590a-4690-9c0a-68827221721a-kube-api-access-lwhn4\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.157357 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.157333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c3713f4-590a-4690-9c0a-68827221721a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.157587 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.157571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.157626 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.157602    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.159318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.159295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c3713f4-590a-4690-9c0a-68827221721a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.166205 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.166182    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhn4\" (UniqueName: \"kubernetes.io/projected/4c3713f4-590a-4690-9c0a-68827221721a-kube-api-access-lwhn4\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.223149 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.223104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:14.364280 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.364251    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"]
Jul 28 13:01:14.366671 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:01:14.366643    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3713f4_590a_4690_9c0a_68827221721a.slice/crio-97bd8a2b5f1f3fdec1077960d546d165d0dfdf8b432a41e9cf081343561f51e1 WatchSource:0}: Error finding container 97bd8a2b5f1f3fdec1077960d546d165d0dfdf8b432a41e9cf081343561f51e1: Status 404 returned error can't find the container with id 97bd8a2b5f1f3fdec1077960d546d165d0dfdf8b432a41e9cf081343561f51e1
Jul 28 13:01:14.860328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.860291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerStarted","Data":"a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b"}
Jul 28 13:01:14.860523 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:14.860382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerStarted","Data":"97bd8a2b5f1f3fdec1077960d546d165d0dfdf8b432a41e9cf081343561f51e1"}
Jul 28 13:01:19.885639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:19.885606    2576 generic.go:356] "Generic (PLEG): container finished" podID="4c3713f4-590a-4690-9c0a-68827221721a" containerID="a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b" exitCode=0
Jul 28 13:01:19.885971 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:19.885682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerDied","Data":"a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b"}
Jul 28 13:01:20.892698 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:20.892663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerStarted","Data":"b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb"}
Jul 28 13:01:20.893119 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:20.892722    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerStarted","Data":"9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a"}
Jul 28 13:01:20.893119 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:20.893002    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:20.893228 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:20.893164    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:20.894262 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:20.894231    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:20.914486 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:20.914446    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" podStartSLOduration=3.351196283 podStartE2EDuration="7.91442892s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.563232637s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:20.912676911 +0000 UTC m=+1573.813821311" watchObservedRunningTime="2026-07-28 13:01:20.91442892 +0000 UTC m=+1573.815573309"
Jul 28 13:01:21.897492 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:21.897456    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:26.902000 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:26.901969    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:26.902679 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:26.902649    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:01:36.903815 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:36.903783    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:48.974500 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:48.974461    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"]
Jul 28 13:01:48.975091 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:48.974693    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container" containerID="cri-o://9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a" gracePeriod=30
Jul 28 13:01:48.975091 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:48.974733    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kube-rbac-proxy" containerID="cri-o://b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb" gracePeriod=30
Jul 28 13:01:50.037320 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.037292    2576 generic.go:356] "Generic (PLEG): container finished" podID="4c3713f4-590a-4690-9c0a-68827221721a" containerID="b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb" exitCode=2
Jul 28 13:01:50.037640 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.037360    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerDied","Data":"b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb"}
Jul 28 13:01:50.218493 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.218469    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:50.338331 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338310    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwhn4\" (UniqueName: \"kubernetes.io/projected/4c3713f4-590a-4690-9c0a-68827221721a-kube-api-access-lwhn4\") pod \"4c3713f4-590a-4690-9c0a-68827221721a\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") "
Jul 28 13:01:50.338496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c3713f4-590a-4690-9c0a-68827221721a-kserve-provision-location\") pod \"4c3713f4-590a-4690-9c0a-68827221721a\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") "
Jul 28 13:01:50.338496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338455    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c3713f4-590a-4690-9c0a-68827221721a-proxy-tls\") pod \"4c3713f4-590a-4690-9c0a-68827221721a\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") "
Jul 28 13:01:50.338626 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338497    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-kube-rbac-proxy-sar-config\") pod \"4c3713f4-590a-4690-9c0a-68827221721a\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") "
Jul 28 13:01:50.338626 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338544    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-cabundle-cert\") pod \"4c3713f4-590a-4690-9c0a-68827221721a\" (UID: \"4c3713f4-590a-4690-9c0a-68827221721a\") "
Jul 28 13:01:50.338918 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338881    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c3713f4-590a-4690-9c0a-68827221721a" (UID: "4c3713f4-590a-4690-9c0a-68827221721a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:50.338918 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.338907    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c3713f4-590a-4690-9c0a-68827221721a" (UID: "4c3713f4-590a-4690-9c0a-68827221721a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:01:50.339472 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.339454    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3713f4-590a-4690-9c0a-68827221721a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c3713f4-590a-4690-9c0a-68827221721a" (UID: "4c3713f4-590a-4690-9c0a-68827221721a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:01:50.340364 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.340344    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3713f4-590a-4690-9c0a-68827221721a-kube-api-access-lwhn4" (OuterVolumeSpecName: "kube-api-access-lwhn4") pod "4c3713f4-590a-4690-9c0a-68827221721a" (UID: "4c3713f4-590a-4690-9c0a-68827221721a"). InnerVolumeSpecName "kube-api-access-lwhn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:01:50.340484 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.340358    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3713f4-590a-4690-9c0a-68827221721a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c3713f4-590a-4690-9c0a-68827221721a" (UID: "4c3713f4-590a-4690-9c0a-68827221721a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:01:50.439998 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.439977    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:50.439998 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.439998    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c3713f4-590a-4690-9c0a-68827221721a-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:50.440130 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.440007    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwhn4\" (UniqueName: \"kubernetes.io/projected/4c3713f4-590a-4690-9c0a-68827221721a-kube-api-access-lwhn4\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:50.440130 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.440015    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c3713f4-590a-4690-9c0a-68827221721a-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:50.440130 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:50.440023    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c3713f4-590a-4690-9c0a-68827221721a-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:01:51.043863 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.043838    2576 generic.go:356] "Generic (PLEG): container finished" podID="4c3713f4-590a-4690-9c0a-68827221721a" containerID="9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a" exitCode=0
Jul 28 13:01:51.044194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.043897    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"
Jul 28 13:01:51.044194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.043921    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerDied","Data":"9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a"}
Jul 28 13:01:51.044194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.043986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k" event={"ID":"4c3713f4-590a-4690-9c0a-68827221721a","Type":"ContainerDied","Data":"97bd8a2b5f1f3fdec1077960d546d165d0dfdf8b432a41e9cf081343561f51e1"}
Jul 28 13:01:51.044194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.044012    2576 scope.go:117] "RemoveContainer" containerID="b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb"
Jul 28 13:01:51.052708 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.052691    2576 scope.go:117] "RemoveContainer" containerID="9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a"
Jul 28 13:01:51.059910 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.059893    2576 scope.go:117] "RemoveContainer" containerID="a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b"
Jul 28 13:01:51.067014 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.066930    2576 scope.go:117] "RemoveContainer" containerID="b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb"
Jul 28 13:01:51.067322 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:51.067300    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb\": container with ID starting with b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb not found: ID does not exist" containerID="b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb"
Jul 28 13:01:51.067428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.067326    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb"} err="failed to get container status \"b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb\": rpc error: code = NotFound desc = could not find container \"b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb\": container with ID starting with b964c162175f4fb24fcf424711288f672b29e170a678c5340cbf9279b57490fb not found: ID does not exist"
Jul 28 13:01:51.067428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.067343    2576 scope.go:117] "RemoveContainer" containerID="9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a"
Jul 28 13:01:51.067611 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:51.067595    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a\": container with ID starting with 9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a not found: ID does not exist" containerID="9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a"
Jul 28 13:01:51.067660 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.067617    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a"} err="failed to get container status \"9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a\": rpc error: code = NotFound desc = could not find container \"9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a\": container with ID starting with 9eb85923dc4c2bd80d0bd5851333214ee057517f5fec5340739b12056495bc2a not found: ID does not exist"
Jul 28 13:01:51.067660 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.067633    2576 scope.go:117] "RemoveContainer" containerID="a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b"
Jul 28 13:01:51.067864 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:51.067851    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b\": container with ID starting with a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b not found: ID does not exist" containerID="a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b"
Jul 28 13:01:51.067913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.067869    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b"} err="failed to get container status \"a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b\": rpc error: code = NotFound desc = could not find container \"a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b\": container with ID starting with a28bc7365feeb46c366a703d11a069212d8189d979d62bac110007c1b706a66b not found: ID does not exist"
Jul 28 13:01:51.068908 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.068889    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"]
Jul 28 13:01:51.073040 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.073021    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vtq8k"]
Jul 28 13:01:51.720243 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:51.720209    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c3713f4-590a-4690-9c0a-68827221721a" path="/var/lib/kubelet/pods/4c3713f4-590a-4690-9c0a-68827221721a/volumes"
Jul 28 13:01:54.157055 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157018    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"]
Jul 28 13:01:54.157536 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157516    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container"
Jul 28 13:01:54.157611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157538    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container"
Jul 28 13:01:54.157611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157562    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="storage-initializer"
Jul 28 13:01:54.157611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157571    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="storage-initializer"
Jul 28 13:01:54.157611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157591    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kube-rbac-proxy"
Jul 28 13:01:54.157611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157599    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kube-rbac-proxy"
Jul 28 13:01:54.157868 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157688    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kserve-container"
Jul 28 13:01:54.157868 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.157703    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3713f4-590a-4690-9c0a-68827221721a" containerName="kube-rbac-proxy"
Jul 28 13:01:54.162661 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.162641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.165626 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.165606    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 13:01:54.165784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.165611    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:01:54.165784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.165636    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:01:54.165784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.165657    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:01:54.165784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.165611    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:01:54.166072 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.165970    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 13:01:54.166639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.166622    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:01:54.173827 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.173806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"]
Jul 28 13:01:54.193947 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.193920    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"]
Jul 28 13:01:54.197504 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.197488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.200259 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.200243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 13:01:54.207408 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.207390    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"]
Jul 28 13:01:54.264446 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcbk\" (UniqueName: \"kubernetes.io/projected/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-api-access-rvcbk\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.264582 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.264582 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.264582 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kserve-provision-location\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.264727 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.264727 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.264727 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jxn\" (UniqueName: \"kubernetes.io/projected/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-api-access-66jxn\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.264838 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.264838 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.264932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.264841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-cabundle-cert\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366068 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366177 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-cabundle-cert\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366177 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcbk\" (UniqueName: \"kubernetes.io/projected/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-api-access-rvcbk\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366177 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366177 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kserve-provision-location\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jxn\" (UniqueName: \"kubernetes.io/projected/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-api-access-66jxn\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366409 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:54.366397    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 13:01:54.366738 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:01:54.366454    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls podName:f36fc980-7685-43aa-ba98-5a8c7dd30a80 nodeName:}" failed. No retries permitted until 2026-07-28 13:01:54.866433245 +0000 UTC m=+1607.767577619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls") pod "isvc-canary-create-predictor-5658fb66fc-shwk4" (UID: "f36fc980-7685-43aa-ba98-5a8c7dd30a80") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 13:01:54.366738 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kserve-provision-location\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366902    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-cabundle-cert\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.366990 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366990 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.366990 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.366963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.367109 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.367006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.368946 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.368928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-proxy-tls\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.375174 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.375156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jxn\" (UniqueName: \"kubernetes.io/projected/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-api-access-66jxn\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.375497 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.375480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcbk\" (UniqueName: \"kubernetes.io/projected/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-api-access-rvcbk\") pod \"isvc-canary-create-v2-predictor-69987fd58d-5t8cz\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.507428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.507361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:01:54.643512 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.643481    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"]
Jul 28 13:01:54.646557 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:01:54.646531    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a704a41_7fe2_4b6a_9460_5fee2c0d7f68.slice/crio-3be9fc7bededbffda56637986bb55dc5ef49bb808804276f08dc0f1965c6a1fe WatchSource:0}: Error finding container 3be9fc7bededbffda56637986bb55dc5ef49bb808804276f08dc0f1965c6a1fe: Status 404 returned error can't find the container with id 3be9fc7bededbffda56637986bb55dc5ef49bb808804276f08dc0f1965c6a1fe
Jul 28 13:01:54.870893 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.870868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:54.873083 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:54.873065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls\") pod \"isvc-canary-create-predictor-5658fb66fc-shwk4\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:55.064347 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:55.064314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerStarted","Data":"7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8"}
Jul 28 13:01:55.064501 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:55.064394    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerStarted","Data":"3be9fc7bededbffda56637986bb55dc5ef49bb808804276f08dc0f1965c6a1fe"}
Jul 28 13:01:55.074543 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:55.074526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:01:55.210441 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:55.210411    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"]
Jul 28 13:01:55.214709 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:01:55.214683    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36fc980_7685_43aa_ba98_5a8c7dd30a80.slice/crio-c02feba657e6573cb5de3bd64d931e699b3c68630c93b1d26f5c10a123da2be1 WatchSource:0}: Error finding container c02feba657e6573cb5de3bd64d931e699b3c68630c93b1d26f5c10a123da2be1: Status 404 returned error can't find the container with id c02feba657e6573cb5de3bd64d931e699b3c68630c93b1d26f5c10a123da2be1
Jul 28 13:01:56.070897 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:56.070860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerStarted","Data":"cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d"}
Jul 28 13:01:56.071072 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:56.070952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerStarted","Data":"c02feba657e6573cb5de3bd64d931e699b3c68630c93b1d26f5c10a123da2be1"}
Jul 28 13:01:59.087100 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:59.087041    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerID="7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8" exitCode=0
Jul 28 13:01:59.087454 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:59.087115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerDied","Data":"7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8"}
Jul 28 13:01:59.088782 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:59.088763    2576 generic.go:356] "Generic (PLEG): container finished" podID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerID="cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d" exitCode=0
Jul 28 13:01:59.088865 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:01:59.088815    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerDied","Data":"cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d"}
Jul 28 13:02:00.096453 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.096407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerStarted","Data":"e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51"}
Jul 28 13:02:00.096920 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.096501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerStarted","Data":"999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019"}
Jul 28 13:02:00.096920 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.096865    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:02:00.097048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.096973    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:02:00.098736 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.098707    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:00.098736 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.098723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerStarted","Data":"d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27"}
Jul 28 13:02:00.098920 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.098772    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerStarted","Data":"376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce"}
Jul 28 13:02:00.098984 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.098966    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:02:00.118089 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.118045    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podStartSLOduration=2.484849137 podStartE2EDuration="6.118032048s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633182911s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:00.11695222 +0000 UTC m=+1613.018096610" watchObservedRunningTime="2026-07-28 13:02:00.118032048 +0000 UTC m=+1613.019176438"
Jul 28 13:02:00.136498 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:00.136455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podStartSLOduration=2.432768473 podStartE2EDuration="6.136441363s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70367289s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:00.135466915 +0000 UTC m=+1613.036611631" watchObservedRunningTime="2026-07-28 13:02:00.136441363 +0000 UTC m=+1613.037585754"
Jul 28 13:02:01.104744 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:01.104708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:02:01.105188 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:01.104740    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:01.106062 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:01.106039    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:02.109227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:02.109182    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:06.109668 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:06.109641    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:02:06.110229 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:06.110199    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:07.114379 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:07.114338    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:02:07.114927 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:07.114900    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:16.110987 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:16.110946    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:17.115901 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:17.115856    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:26.110643 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:26.110604    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:27.115592 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:27.115552    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:36.111039 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:36.110998    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:37.114976 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:37.114936    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:46.111032 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:46.110983    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:47.115490 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:47.115452    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:02:56.110633 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:56.110587    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:02:57.115872 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:02:57.115832    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:03:06.110963 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:06.110920    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:03:07.115308 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:07.115268    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:03:16.111161 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:16.111125    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:03:17.116179 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:17.116152    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:03:24.393328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:24.393286    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"]
Jul 28 13:03:24.393788 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:24.393548    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" containerID="cri-o://999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019" gracePeriod=30
Jul 28 13:03:24.393788 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:24.393592    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kube-rbac-proxy" containerID="cri-o://e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51" gracePeriod=30
Jul 28 13:03:24.412357 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:24.412316    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"]
Jul 28 13:03:24.412672 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:24.412629    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" containerID="cri-o://376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce" gracePeriod=30
Jul 28 13:03:24.412769 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:24.412697    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kube-rbac-proxy" containerID="cri-o://d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27" gracePeriod=30
Jul 28 13:03:25.501552 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:25.501517    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerID="d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27" exitCode=2
Jul 28 13:03:25.501971 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:25.501593    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerDied","Data":"d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27"}
Jul 28 13:03:25.506620 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:25.506586    2576 generic.go:356] "Generic (PLEG): container finished" podID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerID="e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51" exitCode=2
Jul 28 13:03:25.506779 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:25.506657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerDied","Data":"e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51"}
Jul 28 13:03:26.105227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:26.105180    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 13:03:26.110698 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:26.110663    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:03:27.110297 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:27.110256    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 13:03:27.115060 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:27.115030    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:03:29.503916 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.503827    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"]
Jul 28 13:03:29.507805 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.507780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.509794 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.509769    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"]
Jul 28 13:03:29.510800 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.510780    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 13:03:29.510903 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.510787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 13:03:29.513458 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.513440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.515883 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.515860    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 13:03:29.520355 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.520332    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"]
Jul 28 13:03:29.526864 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.526839    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"]
Jul 28 13:03:29.630625 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.630771 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7ng\" (UniqueName: \"kubernetes.io/projected/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-api-access-zz7ng\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.630771 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx44c\" (UniqueName: \"kubernetes.io/projected/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-api-access-fx44c\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.630771 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-cabundle-cert\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.630771 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f97f7d9-e071-476e-bd73-5922d59c5085-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.630958 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.630958 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc38f686-1143-4adc-a09e-74c9e75bac5c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.630958 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.630958 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.631108 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.630976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.732328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-cabundle-cert\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.732555 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f97f7d9-e071-476e-bd73-5922d59c5085-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.732555 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.732555 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc38f686-1143-4adc-a09e-74c9e75bac5c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.732555 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.732555 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.732846 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.732846 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.732846 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7ng\" (UniqueName: \"kubernetes.io/projected/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-api-access-zz7ng\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.732846 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.732700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx44c\" (UniqueName: \"kubernetes.io/projected/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-api-access-fx44c\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.732846 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:29.732806    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 13:03:29.733123 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:29.732898    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls podName:9f97f7d9-e071-476e-bd73-5922d59c5085 nodeName:}" failed. No retries permitted until 2026-07-28 13:03:30.232874532 +0000 UTC m=+1703.134018903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls") pod "isvc-canary-promote-v2-predictor-f59b85f46-f74hc" (UID: "9f97f7d9-e071-476e-bd73-5922d59c5085") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 13:03:29.733123 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:29.733087    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 13:03:29.733249 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:29.733134    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls podName:cc38f686-1143-4adc-a09e-74c9e75bac5c nodeName:}" failed. No retries permitted until 2026-07-28 13:03:30.233118593 +0000 UTC m=+1703.134262961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls") pod "isvc-canary-promote-predictor-7bcdb47df7-74gvf" (UID: "cc38f686-1143-4adc-a09e-74c9e75bac5c") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 13:03:29.733249 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.733172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-cabundle-cert\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.733510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.733342    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.733510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.733490    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc38f686-1143-4adc-a09e-74c9e75bac5c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.733726 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.733699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.733846 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.733791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f97f7d9-e071-476e-bd73-5922d59c5085-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.734115 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.734099    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.744690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.744661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7ng\" (UniqueName: \"kubernetes.io/projected/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-api-access-zz7ng\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:29.744850 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.744704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx44c\" (UniqueName: \"kubernetes.io/projected/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-api-access-fx44c\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:29.775099 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.775068    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:03:29.777104 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.777085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:03:29.833958 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.833923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-cabundle-cert\") pod \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") "
Jul 28 13:03:29.834169 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.833972    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66jxn\" (UniqueName: \"kubernetes.io/projected/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-api-access-66jxn\") pod \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") "
Jul 28 13:03:29.834169 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834011    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kserve-provision-location\") pod \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") "
Jul 28 13:03:29.834289 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834174    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvcbk\" (UniqueName: \"kubernetes.io/projected/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-api-access-rvcbk\") pod \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") "
Jul 28 13:03:29.834289 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834247    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls\") pod \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") "
Jul 28 13:03:29.834428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834285    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-rbac-proxy-sar-config\") pod \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") "
Jul 28 13:03:29.834428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" (UID: "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:03:29.834428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" (UID: "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:03:29.834428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834364    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kserve-provision-location\") pod \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") "
Jul 28 13:03:29.834428 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834415    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-proxy-tls\") pod \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") "
Jul 28 13:03:29.834710 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834472    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-rbac-proxy-sar-config\") pod \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\" (UID: \"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68\") "
Jul 28 13:03:29.834710 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-cabundle-cert\") pod \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\" (UID: \"f36fc980-7685-43aa-ba98-5a8c7dd30a80\") "
Jul 28 13:03:29.834821 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.834737    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f36fc980-7685-43aa-ba98-5a8c7dd30a80" (UID: "f36fc980-7685-43aa-ba98-5a8c7dd30a80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:03:29.835206 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.835111    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" (UID: "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:03:29.835601 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.835134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f36fc980-7685-43aa-ba98-5a8c7dd30a80" (UID: "f36fc980-7685-43aa-ba98-5a8c7dd30a80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:03:29.835601 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.835307    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f36fc980-7685-43aa-ba98-5a8c7dd30a80" (UID: "f36fc980-7685-43aa-ba98-5a8c7dd30a80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836790    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f36fc980-7685-43aa-ba98-5a8c7dd30a80" (UID: "f36fc980-7685-43aa-ba98-5a8c7dd30a80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836797    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-api-access-66jxn" (OuterVolumeSpecName: "kube-api-access-66jxn") pod "f36fc980-7685-43aa-ba98-5a8c7dd30a80" (UID: "f36fc980-7685-43aa-ba98-5a8c7dd30a80"). InnerVolumeSpecName "kube-api-access-66jxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836799    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836835    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836851    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836865    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836880    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836894    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.837048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.836986    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-api-access-rvcbk" (OuterVolumeSpecName: "kube-api-access-rvcbk") pod "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" (UID: "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68"). InnerVolumeSpecName "kube-api-access-rvcbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:03:29.837559 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.837472    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" (UID: "5a704a41-7fe2-4b6a-9460-5fee2c0d7f68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:03:29.938172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.938126    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jxn\" (UniqueName: \"kubernetes.io/projected/f36fc980-7685-43aa-ba98-5a8c7dd30a80-kube-api-access-66jxn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.938172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.938157    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvcbk\" (UniqueName: \"kubernetes.io/projected/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-kube-api-access-rvcbk\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.938172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.938167    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f36fc980-7685-43aa-ba98-5a8c7dd30a80-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:29.938172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:29.938177    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:03:30.240690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.240650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:30.240690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.240690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:30.243223 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.243196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f59b85f46-f74hc\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:30.243299 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.243196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls\") pod \"isvc-canary-promote-predictor-7bcdb47df7-74gvf\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:30.421251 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.421206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:30.428828 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.428798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:30.540483 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.540412    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerID="376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce" exitCode=0
Jul 28 13:03:30.540946 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.540490    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"
Jul 28 13:03:30.540946 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.540503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerDied","Data":"376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce"}
Jul 28 13:03:30.540946 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.540580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz" event={"ID":"5a704a41-7fe2-4b6a-9460-5fee2c0d7f68","Type":"ContainerDied","Data":"3be9fc7bededbffda56637986bb55dc5ef49bb808804276f08dc0f1965c6a1fe"}
Jul 28 13:03:30.540946 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.540625    2576 scope.go:117] "RemoveContainer" containerID="d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27"
Jul 28 13:03:30.544243 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.544050    2576 generic.go:356] "Generic (PLEG): container finished" podID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerID="999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019" exitCode=0
Jul 28 13:03:30.544243 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.544124    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"
Jul 28 13:03:30.544243 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.544144    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerDied","Data":"999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019"}
Jul 28 13:03:30.544243 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.544202    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4" event={"ID":"f36fc980-7685-43aa-ba98-5a8c7dd30a80","Type":"ContainerDied","Data":"c02feba657e6573cb5de3bd64d931e699b3c68630c93b1d26f5c10a123da2be1"}
Jul 28 13:03:30.554025 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.553950    2576 scope.go:117] "RemoveContainer" containerID="376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce"
Jul 28 13:03:30.565502 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.565477    2576 scope.go:117] "RemoveContainer" containerID="7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8"
Jul 28 13:03:30.571834 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.571811    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"]
Jul 28 13:03:30.576081 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.576059    2576 scope.go:117] "RemoveContainer" containerID="d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27"
Jul 28 13:03:30.576942 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.576917    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"]
Jul 28 13:03:30.577071 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:30.577054    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27\": container with ID starting with d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27 not found: ID does not exist" containerID="d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27"
Jul 28 13:03:30.577127 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.577081    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27"} err="failed to get container status \"d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27\": rpc error: code = NotFound desc = could not find container \"d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27\": container with ID starting with d9cb0f54594a43cb98cbbaf828cf126c3f2fe7d1f55a4e3143fb650e9b07ae27 not found: ID does not exist"
Jul 28 13:03:30.577127 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.577100    2576 scope.go:117] "RemoveContainer" containerID="376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce"
Jul 28 13:03:30.577461 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:30.577429    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce\": container with ID starting with 376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce not found: ID does not exist" containerID="376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce"
Jul 28 13:03:30.577566 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.577467    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce"} err="failed to get container status \"376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce\": rpc error: code = NotFound desc = could not find container \"376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce\": container with ID starting with 376e2d5a424aaca88bd4ac170990ea052af97e9c400dd6475cf96ad94d70c5ce not found: ID does not exist"
Jul 28 13:03:30.577566 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.577493    2576 scope.go:117] "RemoveContainer" containerID="7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8"
Jul 28 13:03:30.577739 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:03:30.577719    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc38f686_1143_4adc_a09e_74c9e75bac5c.slice/crio-bb231b96fb4ed0cf6d44d5ea15a7c4589e3a298aa4a7d9e2930c2618ff2670c5 WatchSource:0}: Error finding container bb231b96fb4ed0cf6d44d5ea15a7c4589e3a298aa4a7d9e2930c2618ff2670c5: Status 404 returned error can't find the container with id bb231b96fb4ed0cf6d44d5ea15a7c4589e3a298aa4a7d9e2930c2618ff2670c5
Jul 28 13:03:30.577785 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:30.577742    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8\": container with ID starting with 7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8 not found: ID does not exist" containerID="7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8"
Jul 28 13:03:30.577785 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.577770    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8"} err="failed to get container status \"7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8\": rpc error: code = NotFound desc = could not find container \"7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8\": container with ID starting with 7cc3a34655629882481091768b516884af84b8e14bfc22ce8f9d77a5d5fb02c8 not found: ID does not exist"
Jul 28 13:03:30.577851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.577788    2576 scope.go:117] "RemoveContainer" containerID="e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51"
Jul 28 13:03:30.582527 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.582505    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-69987fd58d-5t8cz"]
Jul 28 13:03:30.587072 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.587046    2576 scope.go:117] "RemoveContainer" containerID="999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019"
Jul 28 13:03:30.596448 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.596414    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"]
Jul 28 13:03:30.600817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.600789    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5658fb66fc-shwk4"]
Jul 28 13:03:30.603176 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.603157    2576 scope.go:117] "RemoveContainer" containerID="cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d"
Jul 28 13:03:30.603617 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.603591    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"]
Jul 28 13:03:30.604590 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:03:30.604567    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f97f7d9_e071_476e_bd73_5922d59c5085.slice/crio-55397431d76f7988ed84004f7de54b3a872f1bd079ec0d97ddb1a5f01ca9926c WatchSource:0}: Error finding container 55397431d76f7988ed84004f7de54b3a872f1bd079ec0d97ddb1a5f01ca9926c: Status 404 returned error can't find the container with id 55397431d76f7988ed84004f7de54b3a872f1bd079ec0d97ddb1a5f01ca9926c
Jul 28 13:03:30.639966 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.639947    2576 scope.go:117] "RemoveContainer" containerID="e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51"
Jul 28 13:03:30.640328 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:30.640278    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51\": container with ID starting with e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51 not found: ID does not exist" containerID="e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51"
Jul 28 13:03:30.640328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.640307    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51"} err="failed to get container status \"e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51\": rpc error: code = NotFound desc = could not find container \"e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51\": container with ID starting with e6f34a7c5bd61556bc93214f32317d07227b5fb0474a6567321ad8bbfd9e3b51 not found: ID does not exist"
Jul 28 13:03:30.640328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.640328    2576 scope.go:117] "RemoveContainer" containerID="999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019"
Jul 28 13:03:30.640614 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:30.640595    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019\": container with ID starting with 999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019 not found: ID does not exist" containerID="999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019"
Jul 28 13:03:30.640679 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.640624    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019"} err="failed to get container status \"999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019\": rpc error: code = NotFound desc = could not find container \"999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019\": container with ID starting with 999ee1fc60d4b3818280772c019ed9cba7980db9fc797692df0dd2d9caedb019 not found: ID does not exist"
Jul 28 13:03:30.640679 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.640649    2576 scope.go:117] "RemoveContainer" containerID="cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d"
Jul 28 13:03:30.640886 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:03:30.640870    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d\": container with ID starting with cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d not found: ID does not exist" containerID="cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d"
Jul 28 13:03:30.640924 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:30.640891    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d"} err="failed to get container status \"cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d\": rpc error: code = NotFound desc = could not find container \"cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d\": container with ID starting with cf4478c794bf165a6a9c21923089c9b69dc7c7b6274bfb9532622839aee5bd3d not found: ID does not exist"
Jul 28 13:03:31.556713 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:31.556672    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerStarted","Data":"24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f"}
Jul 28 13:03:31.557189 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:31.556773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerStarted","Data":"bb231b96fb4ed0cf6d44d5ea15a7c4589e3a298aa4a7d9e2930c2618ff2670c5"}
Jul 28 13:03:31.559110 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:31.559082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerStarted","Data":"1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5"}
Jul 28 13:03:31.559251 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:31.559146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerStarted","Data":"55397431d76f7988ed84004f7de54b3a872f1bd079ec0d97ddb1a5f01ca9926c"}
Jul 28 13:03:31.721686 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:31.721651    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" path="/var/lib/kubelet/pods/5a704a41-7fe2-4b6a-9460-5fee2c0d7f68/volumes"
Jul 28 13:03:31.722152 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:31.722138    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" path="/var/lib/kubelet/pods/f36fc980-7685-43aa-ba98-5a8c7dd30a80/volumes"
Jul 28 13:03:34.577042 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:34.577006    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerID="24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f" exitCode=0
Jul 28 13:03:34.577482 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:34.577081    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerDied","Data":"24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f"}
Jul 28 13:03:34.578978 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:34.578954    2576 generic.go:356] "Generic (PLEG): container finished" podID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerID="1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5" exitCode=0
Jul 28 13:03:34.579099 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:34.579018    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerDied","Data":"1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5"}
Jul 28 13:03:35.587023 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.586987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerStarted","Data":"8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47"}
Jul 28 13:03:35.587490 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.587084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerStarted","Data":"361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb"}
Jul 28 13:03:35.587490 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.587398    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:35.587615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.587547    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:35.589246 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.589216    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:03:35.589598 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.589575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerStarted","Data":"486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233"}
Jul 28 13:03:35.589696 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.589627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerStarted","Data":"82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c"}
Jul 28 13:03:35.589830 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.589813    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:35.612132 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.612077    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podStartSLOduration=3.044646367 podStartE2EDuration="6.612056179s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.567409812s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:03:35.609292207 +0000 UTC m=+1708.510436624" watchObservedRunningTime="2026-07-28 13:03:35.612056179 +0000 UTC m=+1708.513200569"
Jul 28 13:03:35.632754 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:35.632695    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podStartSLOduration=2.929398183 podStartE2EDuration="6.632680321s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.703282138s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:03:35.630884141 +0000 UTC m=+1708.532028528" watchObservedRunningTime="2026-07-28 13:03:35.632680321 +0000 UTC m=+1708.533824712"
Jul 28 13:03:36.596463 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:36.596423    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:03:36.596950 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:36.596689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:36.598043 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:36.598003    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:03:37.602685 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:37.602642    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:03:41.601034 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:41.601005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:03:41.601630 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:41.601593    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:03:42.606826 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:42.606799    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:03:42.607328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:42.607300    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:03:51.601654 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:51.601611    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:03:52.607265 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:03:52.607223    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:01.602227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:01.602187    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:04:02.607786 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:02.607741    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:11.601922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:11.601880    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:04:12.608034 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:12.607990    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:21.602123 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:21.602075    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:04:22.607648 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:22.607600    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:31.602559 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:31.602516    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:04:32.607299 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:32.607245    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:41.602254 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:41.602208    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:04:42.607819 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:42.607735    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:45.716080 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:45.716026    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:04:52.607763 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:52.607715    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:04:55.720575 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:55.720540    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:04:58.716071 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:58.716033    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:04:59.582734 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:59.582700    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"]
Jul 28 13:04:59.582981 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:59.582956    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container" containerID="cri-o://361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb" gracePeriod=30
Jul 28 13:04:59.583073 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:04:59.582990    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kube-rbac-proxy" containerID="cri-o://8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47" gracePeriod=30
Jul 28 13:05:00.008730 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:00.008622    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerID="8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47" exitCode=2
Jul 28 13:05:00.008730 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:00.008701    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerDied","Data":"8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47"}
Jul 28 13:05:01.597503 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:01.597450    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 13:05:05.447441 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.447409    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:05:05.544443 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544360    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-rbac-proxy-sar-config\") pod \"cc38f686-1143-4adc-a09e-74c9e75bac5c\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") "
Jul 28 13:05:05.544677 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-cabundle-cert\") pod \"cc38f686-1143-4adc-a09e-74c9e75bac5c\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") "
Jul 28 13:05:05.544677 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls\") pod \"cc38f686-1143-4adc-a09e-74c9e75bac5c\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") "
Jul 28 13:05:05.544677 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544565    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc38f686-1143-4adc-a09e-74c9e75bac5c-kserve-provision-location\") pod \"cc38f686-1143-4adc-a09e-74c9e75bac5c\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") "
Jul 28 13:05:05.544677 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544669    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz7ng\" (UniqueName: \"kubernetes.io/projected/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-api-access-zz7ng\") pod \"cc38f686-1143-4adc-a09e-74c9e75bac5c\" (UID: \"cc38f686-1143-4adc-a09e-74c9e75bac5c\") "
Jul 28 13:05:05.544974 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544841    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc38f686-1143-4adc-a09e-74c9e75bac5c" (UID: "cc38f686-1143-4adc-a09e-74c9e75bac5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:05.545031 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544970    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc38f686-1143-4adc-a09e-74c9e75bac5c" (UID: "cc38f686-1143-4adc-a09e-74c9e75bac5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:05:05.545031 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.544993    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc38f686-1143-4adc-a09e-74c9e75bac5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc38f686-1143-4adc-a09e-74c9e75bac5c" (UID: "cc38f686-1143-4adc-a09e-74c9e75bac5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:05:05.546838 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.546817    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc38f686-1143-4adc-a09e-74c9e75bac5c" (UID: "cc38f686-1143-4adc-a09e-74c9e75bac5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:05:05.546906 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.546829    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-api-access-zz7ng" (OuterVolumeSpecName: "kube-api-access-zz7ng") pod "cc38f686-1143-4adc-a09e-74c9e75bac5c" (UID: "cc38f686-1143-4adc-a09e-74c9e75bac5c"). InnerVolumeSpecName "kube-api-access-zz7ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:05:05.645653 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.645601    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc38f686-1143-4adc-a09e-74c9e75bac5c-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:05:05.645653 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.645645    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc38f686-1143-4adc-a09e-74c9e75bac5c-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:05:05.645653 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.645658    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz7ng\" (UniqueName: \"kubernetes.io/projected/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-api-access-zz7ng\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:05:05.645653 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.645667    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:05:05.645946 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:05.645677    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc38f686-1143-4adc-a09e-74c9e75bac5c-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:05:06.043823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.043711    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerID="361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb" exitCode=0
Jul 28 13:05:06.043823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.043778    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"
Jul 28 13:05:06.043823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.043793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerDied","Data":"361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb"}
Jul 28 13:05:06.044096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.043857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf" event={"ID":"cc38f686-1143-4adc-a09e-74c9e75bac5c","Type":"ContainerDied","Data":"bb231b96fb4ed0cf6d44d5ea15a7c4589e3a298aa4a7d9e2930c2618ff2670c5"}
Jul 28 13:05:06.044096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.043886    2576 scope.go:117] "RemoveContainer" containerID="8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47"
Jul 28 13:05:06.052933 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.052910    2576 scope.go:117] "RemoveContainer" containerID="361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb"
Jul 28 13:05:06.061803 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.061771    2576 scope.go:117] "RemoveContainer" containerID="24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f"
Jul 28 13:05:06.064894 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.064866    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"]
Jul 28 13:05:06.071401 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.071356    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7bcdb47df7-74gvf"]
Jul 28 13:05:06.072245 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.072228    2576 scope.go:117] "RemoveContainer" containerID="8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47"
Jul 28 13:05:06.072605 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:05:06.072581    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47\": container with ID starting with 8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47 not found: ID does not exist" containerID="8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47"
Jul 28 13:05:06.072719 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.072613    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47"} err="failed to get container status \"8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47\": rpc error: code = NotFound desc = could not find container \"8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47\": container with ID starting with 8bfec669ee6107ca62e4ccb95882ca3f686c912100d88d9360252241afd88f47 not found: ID does not exist"
Jul 28 13:05:06.072719 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.072633    2576 scope.go:117] "RemoveContainer" containerID="361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb"
Jul 28 13:05:06.072940 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:05:06.072921    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb\": container with ID starting with 361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb not found: ID does not exist" containerID="361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb"
Jul 28 13:05:06.072978 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.072946    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb"} err="failed to get container status \"361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb\": rpc error: code = NotFound desc = could not find container \"361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb\": container with ID starting with 361319c9bb56a4022ee5bf162893584f59bd1f3c18d0ad48e2887fb421d2b8eb not found: ID does not exist"
Jul 28 13:05:06.072978 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.072959    2576 scope.go:117] "RemoveContainer" containerID="24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f"
Jul 28 13:05:06.073210 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:05:06.073190    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f\": container with ID starting with 24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f not found: ID does not exist" containerID="24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f"
Jul 28 13:05:06.073270 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:06.073216    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f"} err="failed to get container status \"24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f\": rpc error: code = NotFound desc = could not find container \"24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f\": container with ID starting with 24bd56143437b832b67a54006a7d4d202257f213cfc37402a9d66edd12d6949f not found: ID does not exist"
Jul 28 13:05:07.721584 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:05:07.721544    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" path="/var/lib/kubelet/pods/cc38f686-1143-4adc-a09e-74c9e75bac5c/volumes"
Jul 28 13:06:39.858837 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:39.858745    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"]
Jul 28 13:06:39.861324 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:39.859038    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container" containerID="cri-o://82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c" gracePeriod=30
Jul 28 13:06:39.861324 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:39.859044    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kube-rbac-proxy" containerID="cri-o://486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233" gracePeriod=30
Jul 28 13:06:40.495328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:40.495292    2576 generic.go:356] "Generic (PLEG): container finished" podID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerID="486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233" exitCode=2
Jul 28 13:06:40.495539 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:40.495346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerDied","Data":"486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233"}
Jul 28 13:06:42.603781 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:42.603734    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 13:06:45.004048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004008    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"]
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004396    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004408    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004426    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kube-rbac-proxy"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004432    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kube-rbac-proxy"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004440    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kube-rbac-proxy"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004446    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kube-rbac-proxy"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004459    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004464    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004471    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kube-rbac-proxy"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004476    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kube-rbac-proxy"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004482    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004488    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004498    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="storage-initializer"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004503    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="storage-initializer"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004510    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="storage-initializer"
Jul 28 13:06:45.004510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004515    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="storage-initializer"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004521    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="storage-initializer"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004526    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="storage-initializer"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004577    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kube-rbac-proxy"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004586    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kube-rbac-proxy"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004593    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kserve-container"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004599    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc38f686-1143-4adc-a09e-74c9e75bac5c" containerName="kube-rbac-proxy"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004604    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f36fc980-7685-43aa-ba98-5a8c7dd30a80" containerName="kserve-container"
Jul 28 13:06:45.005082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.004609    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a704a41-7fe2-4b6a-9460-5fee2c0d7f68" containerName="kserve-container"
Jul 28 13:06:45.008093 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.008068    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.010939 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.010913    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 13:06:45.011179 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.011162    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 13:06:45.025006 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.024968    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"]
Jul 28 13:06:45.145409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.145269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0e8db8-abc4-4692-9797-0cf17ed22f9d-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.145817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.145455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgqq\" (UniqueName: \"kubernetes.io/projected/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-api-access-zvgqq\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.145817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.145494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.145817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.145681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.145817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.145750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.246579 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.246530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.246817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.246595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.246817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.246629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0e8db8-abc4-4692-9797-0cf17ed22f9d-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.246817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.246660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgqq\" (UniqueName: \"kubernetes.io/projected/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-api-access-zvgqq\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.246817 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.246676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.247242 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.247216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.247416 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.247394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.247476 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.247442    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.249398 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.249343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0e8db8-abc4-4692-9797-0cf17ed22f9d-proxy-tls\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.255974 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.255896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgqq\" (UniqueName: \"kubernetes.io/projected/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-api-access-zvgqq\") pod \"isvc-canary-rollback-predictor-ff46f797c-qz5jq\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.319517 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.319475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:45.465320 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.465295    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"]
Jul 28 13:06:45.467462 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:06:45.467428    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0e8db8_abc4_4692_9797_0cf17ed22f9d.slice/crio-a254ac432911e670ff1d57bf0c59e4709ebf1e71e83f487b06d7913ea347dafe WatchSource:0}: Error finding container a254ac432911e670ff1d57bf0c59e4709ebf1e71e83f487b06d7913ea347dafe: Status 404 returned error can't find the container with id a254ac432911e670ff1d57bf0c59e4709ebf1e71e83f487b06d7913ea347dafe
Jul 28 13:06:45.519920 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.519876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerStarted","Data":"a254ac432911e670ff1d57bf0c59e4709ebf1e71e83f487b06d7913ea347dafe"}
Jul 28 13:06:45.799177 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.799144    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:06:45.953474 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.953432    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-cabundle-cert\") pod \"9f97f7d9-e071-476e-bd73-5922d59c5085\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") "
Jul 28 13:06:45.953693 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.953501    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx44c\" (UniqueName: \"kubernetes.io/projected/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-api-access-fx44c\") pod \"9f97f7d9-e071-476e-bd73-5922d59c5085\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") "
Jul 28 13:06:45.953693 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.953580    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls\") pod \"9f97f7d9-e071-476e-bd73-5922d59c5085\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") "
Jul 28 13:06:45.953693 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.953609    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f97f7d9-e071-476e-bd73-5922d59c5085-kserve-provision-location\") pod \"9f97f7d9-e071-476e-bd73-5922d59c5085\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") "
Jul 28 13:06:45.953693 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.953629    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-rbac-proxy-sar-config\") pod \"9f97f7d9-e071-476e-bd73-5922d59c5085\" (UID: \"9f97f7d9-e071-476e-bd73-5922d59c5085\") "
Jul 28 13:06:45.954039 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.953905    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f97f7d9-e071-476e-bd73-5922d59c5085" (UID: "9f97f7d9-e071-476e-bd73-5922d59c5085"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:45.954121 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.954024    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f97f7d9-e071-476e-bd73-5922d59c5085-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f97f7d9-e071-476e-bd73-5922d59c5085" (UID: "9f97f7d9-e071-476e-bd73-5922d59c5085"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:06:45.954121 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.954069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f97f7d9-e071-476e-bd73-5922d59c5085" (UID: "9f97f7d9-e071-476e-bd73-5922d59c5085"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:06:45.955836 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.955804    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-api-access-fx44c" (OuterVolumeSpecName: "kube-api-access-fx44c") pod "9f97f7d9-e071-476e-bd73-5922d59c5085" (UID: "9f97f7d9-e071-476e-bd73-5922d59c5085"). InnerVolumeSpecName "kube-api-access-fx44c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:06:45.955941 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:45.955861    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f97f7d9-e071-476e-bd73-5922d59c5085" (UID: "9f97f7d9-e071-476e-bd73-5922d59c5085"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:06:46.054318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.054211    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx44c\" (UniqueName: \"kubernetes.io/projected/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-api-access-fx44c\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:06:46.054318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.054246    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f97f7d9-e071-476e-bd73-5922d59c5085-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:06:46.054318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.054262    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f97f7d9-e071-476e-bd73-5922d59c5085-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:06:46.054318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.054272    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:06:46.054318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.054281    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f97f7d9-e071-476e-bd73-5922d59c5085-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:06:46.527101 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.527061    2576 generic.go:356] "Generic (PLEG): container finished" podID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerID="82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c" exitCode=0
Jul 28 13:06:46.527289 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.527130    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"
Jul 28 13:06:46.527289 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.527141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerDied","Data":"82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c"}
Jul 28 13:06:46.527289 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.527211    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc" event={"ID":"9f97f7d9-e071-476e-bd73-5922d59c5085","Type":"ContainerDied","Data":"55397431d76f7988ed84004f7de54b3a872f1bd079ec0d97ddb1a5f01ca9926c"}
Jul 28 13:06:46.527289 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.527252    2576 scope.go:117] "RemoveContainer" containerID="486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233"
Jul 28 13:06:46.529146 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.529117    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerStarted","Data":"e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90"}
Jul 28 13:06:46.543611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.543583    2576 scope.go:117] "RemoveContainer" containerID="82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c"
Jul 28 13:06:46.552384 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.552343    2576 scope.go:117] "RemoveContainer" containerID="1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5"
Jul 28 13:06:46.562464 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.562436    2576 scope.go:117] "RemoveContainer" containerID="486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233"
Jul 28 13:06:46.562814 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:06:46.562791    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233\": container with ID starting with 486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233 not found: ID does not exist" containerID="486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233"
Jul 28 13:06:46.562887 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.562831    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233"} err="failed to get container status \"486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233\": rpc error: code = NotFound desc = could not find container \"486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233\": container with ID starting with 486406da452101f258772b6eec59f6f545f825578346cfed0531a30d26009233 not found: ID does not exist"
Jul 28 13:06:46.562887 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.562859    2576 scope.go:117] "RemoveContainer" containerID="82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c"
Jul 28 13:06:46.563170 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:06:46.563144    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c\": container with ID starting with 82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c not found: ID does not exist" containerID="82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c"
Jul 28 13:06:46.563227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.563181    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c"} err="failed to get container status \"82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c\": rpc error: code = NotFound desc = could not find container \"82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c\": container with ID starting with 82e93e523291ac2330ebb8cf5047d98373f09e61086ca3994e669e3a3d494f2c not found: ID does not exist"
Jul 28 13:06:46.563227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.563201    2576 scope.go:117] "RemoveContainer" containerID="1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5"
Jul 28 13:06:46.563526 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:06:46.563495    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5\": container with ID starting with 1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5 not found: ID does not exist" containerID="1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5"
Jul 28 13:06:46.563596 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.563525    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5"} err="failed to get container status \"1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5\": rpc error: code = NotFound desc = could not find container \"1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5\": container with ID starting with 1199b8c8673459b69dac9f8f7f8712ecfde87071d02a20efb1910893fe990ec5 not found: ID does not exist"
Jul 28 13:06:46.568917 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.568886    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"]
Jul 28 13:06:46.574694 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:46.574661    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f59b85f46-f74hc"]
Jul 28 13:06:47.722114 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:47.722075    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" path="/var/lib/kubelet/pods/9f97f7d9-e071-476e-bd73-5922d59c5085/volumes"
Jul 28 13:06:49.548514 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:49.548480    2576 generic.go:356] "Generic (PLEG): container finished" podID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerID="e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90" exitCode=0
Jul 28 13:06:49.548936 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:49.548534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerDied","Data":"e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90"}
Jul 28 13:06:50.555625 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:50.555587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerStarted","Data":"9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1"}
Jul 28 13:06:50.556080 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:50.555654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerStarted","Data":"40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b"}
Jul 28 13:06:50.556080 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:50.555857    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:50.576743 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:50.576683    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podStartSLOduration=3.039410972 podStartE2EDuration="6.576664006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.537253034s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:06:50.574308543 +0000 UTC m=+1903.475452933" watchObservedRunningTime="2026-07-28 13:06:50.576664006 +0000 UTC m=+1903.477808395"
Jul 28 13:06:51.560692 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:51.560655    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:51.562117 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:51.562082    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:06:52.565871 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:52.565821    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:06:57.570304 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:57.570267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:06:57.570970 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:06:57.570940    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:07:07.571042 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:07:07.570995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:07:17.571871 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:07:17.571820    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:07:27.571715 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:07:27.571663    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:07:37.571348 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:07:37.571301    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:07:47.571691 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:07:47.571641    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:07:57.571237 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:07:57.571193    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:08:07.571043 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:08:07.570993    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 13:08:12.716550 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:08:12.716506    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:09:35.210851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:35.210757    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"]
Jul 28 13:09:35.211425 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:35.211078    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container" containerID="cri-o://40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b" gracePeriod=30
Jul 28 13:09:35.211425 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:35.211114    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kube-rbac-proxy" containerID="cri-o://9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1" gracePeriod=30
Jul 28 13:09:36.323848 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:36.323797    2576 generic.go:356] "Generic (PLEG): container finished" podID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerID="9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1" exitCode=2
Jul 28 13:09:36.324268 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:36.323871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerDied","Data":"9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1"}
Jul 28 13:09:37.566300 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:37.566253    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 28 13:09:40.436744 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.436696    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"]
Jul 28 13:09:40.437645 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437617    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container"
Jul 28 13:09:40.437645 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437644    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container"
Jul 28 13:09:40.437852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437702    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kube-rbac-proxy"
Jul 28 13:09:40.437852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437712    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kube-rbac-proxy"
Jul 28 13:09:40.437852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437739    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="storage-initializer"
Jul 28 13:09:40.437852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437748    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="storage-initializer"
Jul 28 13:09:40.438048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437897    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kserve-container"
Jul 28 13:09:40.438048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.437911    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f97f7d9-e071-476e-bd73-5922d59c5085" containerName="kube-rbac-proxy"
Jul 28 13:09:40.443329 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.442791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.446056 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.446027    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 13:09:40.446231 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.446129    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 13:09:40.449982 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.449947    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"]
Jul 28 13:09:40.480741 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.480698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.480922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.480754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a402c5-9a2c-43c7-bfa3-f098e349847a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.480922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.480900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a402c5-9a2c-43c7-bfa3-f098e349847a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.481063 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.480961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.481114 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.481083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84wf\" (UniqueName: \"kubernetes.io/projected/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-api-access-p84wf\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.582618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.582574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84wf\" (UniqueName: \"kubernetes.io/projected/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-api-access-p84wf\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.582843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.582656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.582843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.582696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a402c5-9a2c-43c7-bfa3-f098e349847a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.582843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.582727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a402c5-9a2c-43c7-bfa3-f098e349847a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.582843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.582771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.583354 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.583328    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a402c5-9a2c-43c7-bfa3-f098e349847a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.583572 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.583550    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.583648 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.583628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.585709 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.585690    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a402c5-9a2c-43c7-bfa3-f098e349847a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.590995 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.590966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84wf\" (UniqueName: \"kubernetes.io/projected/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-api-access-p84wf\") pod \"isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.756933 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.756823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:40.907544 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:40.907512    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"]
Jul 28 13:09:40.956600 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:09:40.956563    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a402c5_9a2c_43c7_bfa3_f098e349847a.slice/crio-44eb3d35c63a0aca0dc0efe7ac2d44371747819c0fe08bf556d24dcb22880851 WatchSource:0}: Error finding container 44eb3d35c63a0aca0dc0efe7ac2d44371747819c0fe08bf556d24dcb22880851: Status 404 returned error can't find the container with id 44eb3d35c63a0aca0dc0efe7ac2d44371747819c0fe08bf556d24dcb22880851
Jul 28 13:09:41.061918 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.061884    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:09:41.087362 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087329    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-rbac-proxy-sar-config\") pod \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") "
Jul 28 13:09:41.087585 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087407    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-cabundle-cert\") pod \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") "
Jul 28 13:09:41.087668 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087590    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvgqq\" (UniqueName: \"kubernetes.io/projected/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-api-access-zvgqq\") pod \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") "
Jul 28 13:09:41.087737 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087709    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0e8db8-abc4-4692-9797-0cf17ed22f9d-proxy-tls\") pod \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") "
Jul 28 13:09:41.087737 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df0e8db8-abc4-4692-9797-0cf17ed22f9d" (UID: "df0e8db8-abc4-4692-9797-0cf17ed22f9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:41.087871 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087747    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kserve-provision-location\") pod \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\" (UID: \"df0e8db8-abc4-4692-9797-0cf17ed22f9d\") "
Jul 28 13:09:41.087871 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.087775    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df0e8db8-abc4-4692-9797-0cf17ed22f9d" (UID: "df0e8db8-abc4-4692-9797-0cf17ed22f9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:09:41.088072 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.088046    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:09:41.088072 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.088073    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df0e8db8-abc4-4692-9797-0cf17ed22f9d-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:09:41.088448 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.088243    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df0e8db8-abc4-4692-9797-0cf17ed22f9d" (UID: "df0e8db8-abc4-4692-9797-0cf17ed22f9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:09:41.089833 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.089802    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0e8db8-abc4-4692-9797-0cf17ed22f9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df0e8db8-abc4-4692-9797-0cf17ed22f9d" (UID: "df0e8db8-abc4-4692-9797-0cf17ed22f9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:09:41.089934 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.089830    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-api-access-zvgqq" (OuterVolumeSpecName: "kube-api-access-zvgqq") pod "df0e8db8-abc4-4692-9797-0cf17ed22f9d" (UID: "df0e8db8-abc4-4692-9797-0cf17ed22f9d"). InnerVolumeSpecName "kube-api-access-zvgqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:09:41.188994 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.188948    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvgqq\" (UniqueName: \"kubernetes.io/projected/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kube-api-access-zvgqq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:09:41.188994 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.188986    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df0e8db8-abc4-4692-9797-0cf17ed22f9d-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:09:41.188994 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.189000    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df0e8db8-abc4-4692-9797-0cf17ed22f9d-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:09:41.349567 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.349531    2576 generic.go:356] "Generic (PLEG): container finished" podID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerID="40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b" exitCode=0
Jul 28 13:09:41.349784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.349587    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"
Jul 28 13:09:41.349784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.349620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerDied","Data":"40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b"}
Jul 28 13:09:41.349784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.349689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq" event={"ID":"df0e8db8-abc4-4692-9797-0cf17ed22f9d","Type":"ContainerDied","Data":"a254ac432911e670ff1d57bf0c59e4709ebf1e71e83f487b06d7913ea347dafe"}
Jul 28 13:09:41.349784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.349723    2576 scope.go:117] "RemoveContainer" containerID="9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1"
Jul 28 13:09:41.351484 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.351455    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerStarted","Data":"d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020"}
Jul 28 13:09:41.351609 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.351522    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerStarted","Data":"44eb3d35c63a0aca0dc0efe7ac2d44371747819c0fe08bf556d24dcb22880851"}
Jul 28 13:09:41.360143 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.360117    2576 scope.go:117] "RemoveContainer" containerID="40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b"
Jul 28 13:09:41.368675 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.368650    2576 scope.go:117] "RemoveContainer" containerID="e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90"
Jul 28 13:09:41.378131 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.378106    2576 scope.go:117] "RemoveContainer" containerID="9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1"
Jul 28 13:09:41.379049 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:09:41.378626    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1\": container with ID starting with 9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1 not found: ID does not exist" containerID="9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1"
Jul 28 13:09:41.379049 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.378666    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1"} err="failed to get container status \"9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1\": rpc error: code = NotFound desc = could not find container \"9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1\": container with ID starting with 9c400eef076442c46c4ed385bd11482032cc52c1687550aa4135fa318f3229f1 not found: ID does not exist"
Jul 28 13:09:41.379049 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.378695    2576 scope.go:117] "RemoveContainer" containerID="40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b"
Jul 28 13:09:41.379334 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:09:41.379048    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b\": container with ID starting with 40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b not found: ID does not exist" containerID="40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b"
Jul 28 13:09:41.379334 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.379079    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b"} err="failed to get container status \"40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b\": rpc error: code = NotFound desc = could not find container \"40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b\": container with ID starting with 40ae57f6d3a5c2cc40dafbce605da94e8840251461f4b5566d79f5a9ab71578b not found: ID does not exist"
Jul 28 13:09:41.379334 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.379099    2576 scope.go:117] "RemoveContainer" containerID="e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90"
Jul 28 13:09:41.379528 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:09:41.379504    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90\": container with ID starting with e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90 not found: ID does not exist" containerID="e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90"
Jul 28 13:09:41.379586 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.379536    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90"} err="failed to get container status \"e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90\": rpc error: code = NotFound desc = could not find container \"e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90\": container with ID starting with e83ed0fe800e19caaeae3bd76c420d291ef065e21a97f59c8c177d926fbfef90 not found: ID does not exist"
Jul 28 13:09:41.394356 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.394316    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"]
Jul 28 13:09:41.398276 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.398240    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-ff46f797c-qz5jq"]
Jul 28 13:09:41.720753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:41.720668    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" path="/var/lib/kubelet/pods/df0e8db8-abc4-4692-9797-0cf17ed22f9d/volumes"
Jul 28 13:09:45.373815 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:45.373781    2576 generic.go:356] "Generic (PLEG): container finished" podID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerID="d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020" exitCode=0
Jul 28 13:09:45.374247 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:45.373854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerDied","Data":"d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020"}
Jul 28 13:09:46.381247 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:46.381205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerStarted","Data":"a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7"}
Jul 28 13:09:46.381810 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:46.381288    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerStarted","Data":"727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf"}
Jul 28 13:09:46.381810 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:46.381537    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:46.401491 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:46.401425    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podStartSLOduration=2.396534179 podStartE2EDuration="6.401406075s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.004871896s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:09:46.400526112 +0000 UTC m=+2079.301670502" watchObservedRunningTime="2026-07-28 13:09:46.401406075 +0000 UTC m=+2079.302550465"
Jul 28 13:09:47.387708 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:47.387668    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:47.389338 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:47.389297    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:09:48.392417 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:48.392352    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:09:53.397717 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:53.397685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:09:53.398525 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:09:53.398491    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:10:03.399486 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:10:03.399438    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:10:13.398465 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:10:13.398410    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:10:23.398939 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:10:23.398879    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:10:33.398854 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:10:33.398810    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:10:43.398624 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:10:43.398582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:10:53.398591 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:10:53.398542    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:11:03.398687 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:03.398637    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 13:11:13.399809 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:13.399738    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:11:15.479536 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:15.479496    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"]
Jul 28 13:11:15.480155 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:15.479816    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container" containerID="cri-o://727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf" gracePeriod=30
Jul 28 13:11:15.480155 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:15.479870    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kube-rbac-proxy" containerID="cri-o://a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7" gracePeriod=30
Jul 28 13:11:15.795387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:15.795267    2576 generic.go:356] "Generic (PLEG): container finished" podID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerID="a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7" exitCode=2
Jul 28 13:11:15.795387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:15.795335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerDied","Data":"a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7"}
Jul 28 13:11:18.393172 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:18.393119    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 13:11:21.537161 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.537126    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:11:21.647023 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.646972    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a402c5-9a2c-43c7-bfa3-f098e349847a-kserve-provision-location\") pod \"86a402c5-9a2c-43c7-bfa3-f098e349847a\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") "
Jul 28 13:11:21.647219 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647041    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p84wf\" (UniqueName: \"kubernetes.io/projected/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-api-access-p84wf\") pod \"86a402c5-9a2c-43c7-bfa3-f098e349847a\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") "
Jul 28 13:11:21.647219 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647100    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-cabundle-cert\") pod \"86a402c5-9a2c-43c7-bfa3-f098e349847a\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") "
Jul 28 13:11:21.647219 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647127    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a402c5-9a2c-43c7-bfa3-f098e349847a-proxy-tls\") pod \"86a402c5-9a2c-43c7-bfa3-f098e349847a\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") "
Jul 28 13:11:21.647219 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647202    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-rbac-proxy-sar-config\") pod \"86a402c5-9a2c-43c7-bfa3-f098e349847a\" (UID: \"86a402c5-9a2c-43c7-bfa3-f098e349847a\") "
Jul 28 13:11:21.647474 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647421    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86a402c5-9a2c-43c7-bfa3-f098e349847a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86a402c5-9a2c-43c7-bfa3-f098e349847a" (UID: "86a402c5-9a2c-43c7-bfa3-f098e349847a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:11:21.647541 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647490    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86a402c5-9a2c-43c7-bfa3-f098e349847a-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:11:21.647541 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647502    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86a402c5-9a2c-43c7-bfa3-f098e349847a" (UID: "86a402c5-9a2c-43c7-bfa3-f098e349847a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:21.647711 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.647688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86a402c5-9a2c-43c7-bfa3-f098e349847a" (UID: "86a402c5-9a2c-43c7-bfa3-f098e349847a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:11:21.649416 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.649393    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a402c5-9a2c-43c7-bfa3-f098e349847a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86a402c5-9a2c-43c7-bfa3-f098e349847a" (UID: "86a402c5-9a2c-43c7-bfa3-f098e349847a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:11:21.649510 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.649485    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-api-access-p84wf" (OuterVolumeSpecName: "kube-api-access-p84wf") pod "86a402c5-9a2c-43c7-bfa3-f098e349847a" (UID: "86a402c5-9a2c-43c7-bfa3-f098e349847a"). InnerVolumeSpecName "kube-api-access-p84wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:11:21.748854 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.748812    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:11:21.748854 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.748851    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84wf\" (UniqueName: \"kubernetes.io/projected/86a402c5-9a2c-43c7-bfa3-f098e349847a-kube-api-access-p84wf\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:11:21.748854 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.748862    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86a402c5-9a2c-43c7-bfa3-f098e349847a-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:11:21.749085 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.748873    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a402c5-9a2c-43c7-bfa3-f098e349847a-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:11:21.823604 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.823540    2576 generic.go:356] "Generic (PLEG): container finished" podID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerID="727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf" exitCode=0
Jul 28 13:11:21.823929 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.823632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerDied","Data":"727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf"}
Jul 28 13:11:21.823929 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.823663    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"
Jul 28 13:11:21.823929 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.823733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87" event={"ID":"86a402c5-9a2c-43c7-bfa3-f098e349847a","Type":"ContainerDied","Data":"44eb3d35c63a0aca0dc0efe7ac2d44371747819c0fe08bf556d24dcb22880851"}
Jul 28 13:11:21.823929 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.823781    2576 scope.go:117] "RemoveContainer" containerID="a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7"
Jul 28 13:11:21.837249 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.837220    2576 scope.go:117] "RemoveContainer" containerID="727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf"
Jul 28 13:11:21.843483 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.843452    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"]
Jul 28 13:11:21.846557 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.846524    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-c6cdc95b4-j5k87"]
Jul 28 13:11:21.847478 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.847459    2576 scope.go:117] "RemoveContainer" containerID="d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020"
Jul 28 13:11:21.856395 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.856354    2576 scope.go:117] "RemoveContainer" containerID="a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7"
Jul 28 13:11:21.856727 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:11:21.856704    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7\": container with ID starting with a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7 not found: ID does not exist" containerID="a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7"
Jul 28 13:11:21.856811 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.856736    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7"} err="failed to get container status \"a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7\": rpc error: code = NotFound desc = could not find container \"a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7\": container with ID starting with a54ed18981c225b247de5301b6f50ce75d996c1a34284a548615cebdab00bbe7 not found: ID does not exist"
Jul 28 13:11:21.856811 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.856757    2576 scope.go:117] "RemoveContainer" containerID="727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf"
Jul 28 13:11:21.857024 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:11:21.857001    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf\": container with ID starting with 727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf not found: ID does not exist" containerID="727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf"
Jul 28 13:11:21.857083 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.857032    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf"} err="failed to get container status \"727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf\": rpc error: code = NotFound desc = could not find container \"727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf\": container with ID starting with 727e44edb57c0be3077f9fb0cd526f5444b4fc2528958e8bd65d11475d6a8cdf not found: ID does not exist"
Jul 28 13:11:21.857083 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.857049    2576 scope.go:117] "RemoveContainer" containerID="d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020"
Jul 28 13:11:21.857275 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:11:21.857258    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020\": container with ID starting with d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020 not found: ID does not exist" containerID="d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020"
Jul 28 13:11:21.857336 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:21.857278    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020"} err="failed to get container status \"d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020\": rpc error: code = NotFound desc = could not find container \"d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020\": container with ID starting with d9a94c9c97d6997fb75d04c9fdea4e409870ab1bfa2e8f15365b272e6d22b020 not found: ID does not exist"
Jul 28 13:11:23.720076 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:23.720035    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" path="/var/lib/kubelet/pods/86a402c5-9a2c-43c7-bfa3-f098e349847a/volumes"
Jul 28 13:11:25.756048 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756012    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"]
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756421    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756436    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756448    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="storage-initializer"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756454    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="storage-initializer"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756482    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756488    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756493    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kube-rbac-proxy"
Jul 28 13:11:25.756496 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756498    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kube-rbac-proxy"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756505    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kube-rbac-proxy"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756510    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kube-rbac-proxy"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756516    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="storage-initializer"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756522    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="storage-initializer"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756595    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kserve-container"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756603    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a402c5-9a2c-43c7-bfa3-f098e349847a" containerName="kube-rbac-proxy"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756617    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kserve-container"
Jul 28 13:11:25.756759 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.756622    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0e8db8-abc4-4692-9797-0cf17ed22f9d" containerName="kube-rbac-proxy"
Jul 28 13:11:25.761535 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.761510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.764194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.764167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 13:11:25.765716 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.765651    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:11:25.765918 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.765883    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:11:25.765995 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.765891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 13:11:25.766993 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.766962    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:11:25.766993 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.766985    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:11:25.767160 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.767053    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:11:25.773186 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.773160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"]
Jul 28 13:11:25.888480 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.888429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrjt\" (UniqueName: \"kubernetes.io/projected/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-api-access-mqrjt\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.888685 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.888535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.888753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.888682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf03ec8a-0bc2-4729-a734-4861ad583ea5-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.888811 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.888754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.888871 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.888844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.989614 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.989570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf03ec8a-0bc2-4729-a734-4861ad583ea5-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.989802 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.989630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.989802 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.989671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.989802 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.989699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrjt\" (UniqueName: \"kubernetes.io/projected/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-api-access-mqrjt\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.989802 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.989728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.990294 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.990268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.990565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.990543    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.990565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.990555    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.992382 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.992344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf03ec8a-0bc2-4729-a734-4861ad583ea5-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:25.999470 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:25.999436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrjt\" (UniqueName: \"kubernetes.io/projected/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-api-access-mqrjt\") pod \"isvc-lightgbm-predictor-54f85ddf88-r6dgw\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:26.073141 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:26.073014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:26.218747 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:26.218718    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"]
Jul 28 13:11:26.221608 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:11:26.221573    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf03ec8a_0bc2_4729_a734_4861ad583ea5.slice/crio-e0904aecc6c6a4dcf48e8728a8c0027c106360f97fafc44b68c192e0c7820ab2 WatchSource:0}: Error finding container e0904aecc6c6a4dcf48e8728a8c0027c106360f97fafc44b68c192e0c7820ab2: Status 404 returned error can't find the container with id e0904aecc6c6a4dcf48e8728a8c0027c106360f97fafc44b68c192e0c7820ab2
Jul 28 13:11:26.849161 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:26.849120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerStarted","Data":"077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489"}
Jul 28 13:11:26.849814 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:26.849191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerStarted","Data":"e0904aecc6c6a4dcf48e8728a8c0027c106360f97fafc44b68c192e0c7820ab2"}
Jul 28 13:11:30.870173 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:30.870142    2576 generic.go:356] "Generic (PLEG): container finished" podID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerID="077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489" exitCode=0
Jul 28 13:11:30.870746 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:30.870224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerDied","Data":"077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489"}
Jul 28 13:11:30.871645 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:30.871626    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:11:38.919028 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:38.918989    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerStarted","Data":"86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8"}
Jul 28 13:11:38.919517 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:38.919087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerStarted","Data":"79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6"}
Jul 28 13:11:38.919517 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:38.919293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:38.940364 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:38.940299    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podStartSLOduration=2.59069594 podStartE2EDuration="13.940281006s" totalImagesPullingTime="7.577053429s" totalInitContainerRuntime="3.772531637s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:11:38.937707525 +0000 UTC m=+2191.838851916" watchObservedRunningTime="2026-07-28 13:11:38.940281006 +0000 UTC m=+2191.841425396"
Jul 28 13:11:39.923995 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:39.923877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:39.946608 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:39.925386    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:11:40.928458 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:40.928416    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:11:45.933486 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:45.933438    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:11:45.934028 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:45.933999    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:11:55.934655 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:11:55.934607    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:12:05.934273 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:12:05.934224    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:12:15.934888 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:12:15.934840    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:12:25.934079 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:12:25.934038    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:12:35.934188 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:12:35.934109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:12:45.934821 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:12:45.934781    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:12:55.934512 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:12:55.934482    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:13:05.888998 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:05.888961    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"]
Jul 28 13:13:05.889460 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:05.889218    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" containerID="cri-o://79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6" gracePeriod=30
Jul 28 13:13:05.889549 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:05.889474    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kube-rbac-proxy" containerID="cri-o://86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8" gracePeriod=30
Jul 28 13:13:05.929358 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:05.929313    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 28 13:13:05.934179 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:05.934148    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:13:06.311412 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:06.311312    2576 generic.go:356] "Generic (PLEG): container finished" podID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerID="86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8" exitCode=2
Jul 28 13:13:06.311412 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:06.311399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerDied","Data":"86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8"}
Jul 28 13:13:10.929334 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:10.929294    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 28 13:13:11.055235 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.055203    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:13:11.059339 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059309    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"]
Jul 28 13:13:11.059863 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059846    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="storage-initializer"
Jul 28 13:13:11.059919 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059865    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="storage-initializer"
Jul 28 13:13:11.059919 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059891    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container"
Jul 28 13:13:11.059919 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059897    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container"
Jul 28 13:13:11.059919 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059905    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kube-rbac-proxy"
Jul 28 13:13:11.059919 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059911    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kube-rbac-proxy"
Jul 28 13:13:11.060076 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059962    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kube-rbac-proxy"
Jul 28 13:13:11.060076 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.059970    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerName="kserve-container"
Jul 28 13:13:11.062965 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.062947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.065616 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.065591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 13:13:11.065616 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.065608    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:13:11.074060 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.074031    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"]
Jul 28 13:13:11.124449 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124415    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kserve-provision-location\") pod \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") "
Jul 28 13:13:11.124618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124477    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-cabundle-cert\") pod \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") "
Jul 28 13:13:11.124618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124540    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-rbac-proxy-sar-config\") pod \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") "
Jul 28 13:13:11.124618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124570    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqrjt\" (UniqueName: \"kubernetes.io/projected/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-api-access-mqrjt\") pod \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") "
Jul 28 13:13:11.124755 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124639    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf03ec8a-0bc2-4729-a734-4861ad583ea5-proxy-tls\") pod \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\" (UID: \"cf03ec8a-0bc2-4729-a734-4861ad583ea5\") "
Jul 28 13:13:11.124819 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124792    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.124879 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04701160-1d2a-4cf0-92b5-7f07680cc6af-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.124879 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124835    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf03ec8a-0bc2-4729-a734-4861ad583ea5" (UID: "cf03ec8a-0bc2-4729-a734-4861ad583ea5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:13:11.124879 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.125042 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124915    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf03ec8a-0bc2-4729-a734-4861ad583ea5" (UID: "cf03ec8a-0bc2-4729-a734-4861ad583ea5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:13:11.125042 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124926    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf03ec8a-0bc2-4729-a734-4861ad583ea5" (UID: "cf03ec8a-0bc2-4729-a734-4861ad583ea5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:13:11.125042 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.124990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rwn\" (UniqueName: \"kubernetes.io/projected/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-api-access-q7rwn\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.125042 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.125024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.125195 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.125089    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:13:11.125195 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.125102    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:13:11.125195 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.125117    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:13:11.126837 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.126788    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf03ec8a-0bc2-4729-a734-4861ad583ea5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf03ec8a-0bc2-4729-a734-4861ad583ea5" (UID: "cf03ec8a-0bc2-4729-a734-4861ad583ea5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:13:11.126837 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.126810    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-api-access-mqrjt" (OuterVolumeSpecName: "kube-api-access-mqrjt") pod "cf03ec8a-0bc2-4729-a734-4861ad583ea5" (UID: "cf03ec8a-0bc2-4729-a734-4861ad583ea5"). InnerVolumeSpecName "kube-api-access-mqrjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:13:11.225731 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.225621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rwn\" (UniqueName: \"kubernetes.io/projected/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-api-access-q7rwn\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.225731 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.225687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.225941 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.225854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.225941 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.225902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04701160-1d2a-4cf0-92b5-7f07680cc6af-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.225941 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.225933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.226191 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.226009    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrjt\" (UniqueName: \"kubernetes.io/projected/cf03ec8a-0bc2-4729-a734-4861ad583ea5-kube-api-access-mqrjt\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:13:11.226191 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.226040    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf03ec8a-0bc2-4729-a734-4861ad583ea5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:13:11.226191 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:13:11.226141    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 13:13:11.226357 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:13:11.226213    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls podName:04701160-1d2a-4cf0-92b5-7f07680cc6af nodeName:}" failed. No retries permitted until 2026-07-28 13:13:11.726193897 +0000 UTC m=+2284.627338266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" (UID: "04701160-1d2a-4cf0-92b5-7f07680cc6af") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 13:13:11.226357 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.226341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04701160-1d2a-4cf0-92b5-7f07680cc6af-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.226497 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.226345    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.226536 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.226495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.245006 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.244978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rwn\" (UniqueName: \"kubernetes.io/projected/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-api-access-q7rwn\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.334327 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.334292    2576 generic.go:356] "Generic (PLEG): container finished" podID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" containerID="79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6" exitCode=0
Jul 28 13:13:11.334543 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.334364    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"
Jul 28 13:13:11.334543 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.334402    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerDied","Data":"79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6"}
Jul 28 13:13:11.334543 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.334508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw" event={"ID":"cf03ec8a-0bc2-4729-a734-4861ad583ea5","Type":"ContainerDied","Data":"e0904aecc6c6a4dcf48e8728a8c0027c106360f97fafc44b68c192e0c7820ab2"}
Jul 28 13:13:11.334713 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.334546    2576 scope.go:117] "RemoveContainer" containerID="86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8"
Jul 28 13:13:11.344024 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.344008    2576 scope.go:117] "RemoveContainer" containerID="79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6"
Jul 28 13:13:11.351819 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.351798    2576 scope.go:117] "RemoveContainer" containerID="077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489"
Jul 28 13:13:11.359821 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.359799    2576 scope.go:117] "RemoveContainer" containerID="86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8"
Jul 28 13:13:11.360110 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:13:11.360087    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8\": container with ID starting with 86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8 not found: ID does not exist" containerID="86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8"
Jul 28 13:13:11.360194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.360125    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8"} err="failed to get container status \"86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8\": rpc error: code = NotFound desc = could not find container \"86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8\": container with ID starting with 86d583c26dfa6c01bb0300cc588bfddc7c75eb8c711681cafdd35b0a6aa9dda8 not found: ID does not exist"
Jul 28 13:13:11.360194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.360153    2576 scope.go:117] "RemoveContainer" containerID="79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6"
Jul 28 13:13:11.360462 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:13:11.360439    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6\": container with ID starting with 79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6 not found: ID does not exist" containerID="79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6"
Jul 28 13:13:11.360544 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.360470    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6"} err="failed to get container status \"79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6\": rpc error: code = NotFound desc = could not find container \"79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6\": container with ID starting with 79e41ab87dbcd10995b3a016c6cfea694c78eae959c45a4a0756df3ecfa6a2e6 not found: ID does not exist"
Jul 28 13:13:11.360544 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.360487    2576 scope.go:117] "RemoveContainer" containerID="077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489"
Jul 28 13:13:11.360747 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:13:11.360728    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489\": container with ID starting with 077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489 not found: ID does not exist" containerID="077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489"
Jul 28 13:13:11.360785 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.360751    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489"} err="failed to get container status \"077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489\": rpc error: code = NotFound desc = could not find container \"077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489\": container with ID starting with 077610fc1f3d8bf85fed376fb29e102c052da8f2ee4d560c43b94c2b0b72b489 not found: ID does not exist"
Jul 28 13:13:11.370780 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.370749    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"]
Jul 28 13:13:11.388200 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.388168    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-r6dgw"]
Jul 28 13:13:11.720863 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.720828    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf03ec8a-0bc2-4729-a734-4861ad583ea5" path="/var/lib/kubelet/pods/cf03ec8a-0bc2-4729-a734-4861ad583ea5/volumes"
Jul 28 13:13:11.731014 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.730983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.733506 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.733481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:11.974035 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:11.973943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:12.113745 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:12.113666    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"]
Jul 28 13:13:12.116497 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:13:12.116473    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04701160_1d2a_4cf0_92b5_7f07680cc6af.slice/crio-b0d60c006a0130a55f3d1ea544eacddc4ef0d236018f0e3d0218d78c5169fbcb WatchSource:0}: Error finding container b0d60c006a0130a55f3d1ea544eacddc4ef0d236018f0e3d0218d78c5169fbcb: Status 404 returned error can't find the container with id b0d60c006a0130a55f3d1ea544eacddc4ef0d236018f0e3d0218d78c5169fbcb
Jul 28 13:13:12.341905 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:12.341865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerStarted","Data":"16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757"}
Jul 28 13:13:12.342108 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:12.341942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerStarted","Data":"b0d60c006a0130a55f3d1ea544eacddc4ef0d236018f0e3d0218d78c5169fbcb"}
Jul 28 13:13:16.362297 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:16.362264    2576 generic.go:356] "Generic (PLEG): container finished" podID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerID="16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757" exitCode=0
Jul 28 13:13:16.362712 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:16.362337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerDied","Data":"16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757"}
Jul 28 13:13:17.369497 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:17.369460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerStarted","Data":"8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2"}
Jul 28 13:13:17.369981 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:17.369569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerStarted","Data":"c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1"}
Jul 28 13:13:17.369981 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:17.369796    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:17.391393 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:17.391321    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podStartSLOduration=2.88734425 podStartE2EDuration="6.391287357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.503943107s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:13:17.390216508 +0000 UTC m=+2290.291360903" watchObservedRunningTime="2026-07-28 13:13:17.391287357 +0000 UTC m=+2290.292431746"
Jul 28 13:13:18.373935 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:18.373908    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:18.375421 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:18.375396    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:13:19.378592 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:19.378541    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:13:24.383133 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:24.383100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:13:24.383825 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:24.383794    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:13:34.383935 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:34.383891    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:13:44.384408 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:44.384351    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:13:54.384333 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:13:54.384292    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:14:04.384085 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:04.384043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:14:14.384720 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:14.384677    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:14:24.384514 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:24.384467    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:14:31.715459 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:31.715418    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:14:41.719867 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:41.719840    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:14:51.472682 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:51.472648    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"]
Jul 28 13:14:51.473105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:51.472889    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" containerID="cri-o://c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1" gracePeriod=30
Jul 28 13:14:51.473105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:51.472949    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kube-rbac-proxy" containerID="cri-o://8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2" gracePeriod=30
Jul 28 13:14:51.715729 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:51.715688    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:14:51.796125 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:51.796094    2576 generic.go:356] "Generic (PLEG): container finished" podID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerID="8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2" exitCode=2
Jul 28 13:14:51.796290 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:51.796168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerDied","Data":"8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2"}
Jul 28 13:14:54.379002 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:54.378947    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 28 13:14:56.430587 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.430557    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:14:56.567431 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04701160-1d2a-4cf0-92b5-7f07680cc6af-kserve-provision-location\") pod \"04701160-1d2a-4cf0-92b5-7f07680cc6af\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") "
Jul 28 13:14:56.567431 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-rbac-proxy-sar-config\") pod \"04701160-1d2a-4cf0-92b5-7f07680cc6af\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") "
Jul 28 13:14:56.567658 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567484    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7rwn\" (UniqueName: \"kubernetes.io/projected/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-api-access-q7rwn\") pod \"04701160-1d2a-4cf0-92b5-7f07680cc6af\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") "
Jul 28 13:14:56.567658 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567567    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls\") pod \"04701160-1d2a-4cf0-92b5-7f07680cc6af\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") "
Jul 28 13:14:56.567658 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567592    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-cabundle-cert\") pod \"04701160-1d2a-4cf0-92b5-7f07680cc6af\" (UID: \"04701160-1d2a-4cf0-92b5-7f07680cc6af\") "
Jul 28 13:14:56.567943 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04701160-1d2a-4cf0-92b5-7f07680cc6af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04701160-1d2a-4cf0-92b5-7f07680cc6af" (UID: "04701160-1d2a-4cf0-92b5-7f07680cc6af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:14:56.567943 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.567816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04701160-1d2a-4cf0-92b5-7f07680cc6af" (UID: "04701160-1d2a-4cf0-92b5-7f07680cc6af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:14:56.568214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.568167    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04701160-1d2a-4cf0-92b5-7f07680cc6af" (UID: "04701160-1d2a-4cf0-92b5-7f07680cc6af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:14:56.569831 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.569804    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-api-access-q7rwn" (OuterVolumeSpecName: "kube-api-access-q7rwn") pod "04701160-1d2a-4cf0-92b5-7f07680cc6af" (UID: "04701160-1d2a-4cf0-92b5-7f07680cc6af"). InnerVolumeSpecName "kube-api-access-q7rwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:14:56.569831 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.569816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04701160-1d2a-4cf0-92b5-7f07680cc6af" (UID: "04701160-1d2a-4cf0-92b5-7f07680cc6af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:14:56.632229 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632194    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"]
Jul 28 13:14:56.632557 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632545    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kube-rbac-proxy"
Jul 28 13:14:56.632615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632559    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kube-rbac-proxy"
Jul 28 13:14:56.632615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632575    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="storage-initializer"
Jul 28 13:14:56.632615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632580    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="storage-initializer"
Jul 28 13:14:56.632615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632601    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container"
Jul 28 13:14:56.632615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632606    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container"
Jul 28 13:14:56.632799 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632657    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kube-rbac-proxy"
Jul 28 13:14:56.632799 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.632664    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerName="kserve-container"
Jul 28 13:14:56.636112 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.636087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.638689 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.638658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 13:14:56.638831 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.638740    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:14:56.646714 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.646683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"]
Jul 28 13:14:56.669045 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.669010    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04701160-1d2a-4cf0-92b5-7f07680cc6af-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:14:56.669045 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.669039    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:14:56.669045 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.669048    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7rwn\" (UniqueName: \"kubernetes.io/projected/04701160-1d2a-4cf0-92b5-7f07680cc6af-kube-api-access-q7rwn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:14:56.669275 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.669058    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04701160-1d2a-4cf0-92b5-7f07680cc6af-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:14:56.669275 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.669068    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04701160-1d2a-4cf0-92b5-7f07680cc6af-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:14:56.769731 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.769692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383af136-d253-43a6-9d8b-f2548964f684-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.769731 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.769737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383af136-d253-43a6-9d8b-f2548964f684-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.769969 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.769789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.769969 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.769843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.769969 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.769880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67m4q\" (UniqueName: \"kubernetes.io/projected/383af136-d253-43a6-9d8b-f2548964f684-kube-api-access-67m4q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.819194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.819105    2576 generic.go:356] "Generic (PLEG): container finished" podID="04701160-1d2a-4cf0-92b5-7f07680cc6af" containerID="c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1" exitCode=0
Jul 28 13:14:56.819194 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.819168    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"
Jul 28 13:14:56.819402 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.819191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerDied","Data":"c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1"}
Jul 28 13:14:56.819402 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.819249    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg" event={"ID":"04701160-1d2a-4cf0-92b5-7f07680cc6af","Type":"ContainerDied","Data":"b0d60c006a0130a55f3d1ea544eacddc4ef0d236018f0e3d0218d78c5169fbcb"}
Jul 28 13:14:56.819402 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.819276    2576 scope.go:117] "RemoveContainer" containerID="8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2"
Jul 28 13:14:56.827993 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.827976    2576 scope.go:117] "RemoveContainer" containerID="c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1"
Jul 28 13:14:56.835761 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.835736    2576 scope.go:117] "RemoveContainer" containerID="16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757"
Jul 28 13:14:56.842361 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.842337    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"]
Jul 28 13:14:56.844923 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.844905    2576 scope.go:117] "RemoveContainer" containerID="8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2"
Jul 28 13:14:56.845211 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:14:56.845192    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2\": container with ID starting with 8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2 not found: ID does not exist" containerID="8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2"
Jul 28 13:14:56.845269 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.845221    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2"} err="failed to get container status \"8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2\": rpc error: code = NotFound desc = could not find container \"8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2\": container with ID starting with 8ffb9a44d8609358b46eae6a2c6de9b4e8fabfeacf9565a86821e840e738d5e2 not found: ID does not exist"
Jul 28 13:14:56.845269 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.845240    2576 scope.go:117] "RemoveContainer" containerID="c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1"
Jul 28 13:14:56.845508 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:14:56.845488    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1\": container with ID starting with c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1 not found: ID does not exist" containerID="c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1"
Jul 28 13:14:56.845601 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.845518    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1"} err="failed to get container status \"c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1\": rpc error: code = NotFound desc = could not find container \"c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1\": container with ID starting with c64da5f74bf09b32ba6d92dcd78b5e0ad099ee6a2c80e714988ce517034202f1 not found: ID does not exist"
Jul 28 13:14:56.845601 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.845542    2576 scope.go:117] "RemoveContainer" containerID="16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757"
Jul 28 13:14:56.845823 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:14:56.845802    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757\": container with ID starting with 16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757 not found: ID does not exist" containerID="16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757"
Jul 28 13:14:56.845881 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.845832    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757"} err="failed to get container status \"16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757\": rpc error: code = NotFound desc = could not find container \"16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757\": container with ID starting with 16ed940e0df7681e70a8e085a64da8b4b2f34122ea5c53182b84bfb6cba1c757 not found: ID does not exist"
Jul 28 13:14:56.846333 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.846312    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-n59jg"]
Jul 28 13:14:56.870782 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.870743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.870782 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.870779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67m4q\" (UniqueName: \"kubernetes.io/projected/383af136-d253-43a6-9d8b-f2548964f684-kube-api-access-67m4q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.871036 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.870835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383af136-d253-43a6-9d8b-f2548964f684-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.871036 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.870864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383af136-d253-43a6-9d8b-f2548964f684-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.871036 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.870919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.871307 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.871285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383af136-d253-43a6-9d8b-f2548964f684-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.871573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.871548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.871666 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.871591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.873281 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.873255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383af136-d253-43a6-9d8b-f2548964f684-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.879690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.879665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67m4q\" (UniqueName: \"kubernetes.io/projected/383af136-d253-43a6-9d8b-f2548964f684-kube-api-access-67m4q\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:56.948038 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:56.947997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:14:57.081853 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:57.081827    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"]
Jul 28 13:14:57.083760 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:14:57.083721    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383af136_d253_43a6_9d8b_f2548964f684.slice/crio-cc083bc0d09c58ce55ab00b09097cbd8117944ea9bc9c9b08619dd48c7c7d841 WatchSource:0}: Error finding container cc083bc0d09c58ce55ab00b09097cbd8117944ea9bc9c9b08619dd48c7c7d841: Status 404 returned error can't find the container with id cc083bc0d09c58ce55ab00b09097cbd8117944ea9bc9c9b08619dd48c7c7d841
Jul 28 13:14:57.721516 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:57.721475    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04701160-1d2a-4cf0-92b5-7f07680cc6af" path="/var/lib/kubelet/pods/04701160-1d2a-4cf0-92b5-7f07680cc6af/volumes"
Jul 28 13:14:57.826229 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:57.826194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerStarted","Data":"e98217477af27a33e2fa93d27c98b7a2e78e5be964403dda6aa4f46d4d4f3c42"}
Jul 28 13:14:57.826426 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:14:57.826253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerStarted","Data":"cc083bc0d09c58ce55ab00b09097cbd8117944ea9bc9c9b08619dd48c7c7d841"}
Jul 28 13:15:01.845265 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:15:01.845229    2576 generic.go:356] "Generic (PLEG): container finished" podID="383af136-d253-43a6-9d8b-f2548964f684" containerID="e98217477af27a33e2fa93d27c98b7a2e78e5be964403dda6aa4f46d4d4f3c42" exitCode=0
Jul 28 13:15:01.845696 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:15:01.845301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerDied","Data":"e98217477af27a33e2fa93d27c98b7a2e78e5be964403dda6aa4f46d4d4f3c42"}
Jul 28 13:17:11.539168 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:11.539127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerStarted","Data":"5101f3eb61151b27332f7c7b041bf88589b338aba17347c5646503a891077052"}
Jul 28 13:17:11.539603 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:11.539229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerStarted","Data":"a307cce0a8a75b798a0be562d07b96ee45105608e83a5a08ffa8748d05b6cd80"}
Jul 28 13:17:11.539603 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:11.539264    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:17:11.539603 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:11.539291    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:17:11.568576 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:11.568522    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" podStartSLOduration=2.672091858 podStartE2EDuration="2m15.568484805s" totalImagesPullingTime="2m9.077683403s" totalInitContainerRuntime="3.818709544s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:17:11.56832112 +0000 UTC m=+2524.469465511" watchObservedRunningTime="2026-07-28 13:17:11.568484805 +0000 UTC m=+2524.469629196"
Jul 28 13:17:17.551090 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:17.551065    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:17:47.555557 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:47.555524    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:17:56.820218 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:56.820185    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"]
Jul 28 13:17:56.820741 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:56.820426    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kserve-container" containerID="cri-o://a307cce0a8a75b798a0be562d07b96ee45105608e83a5a08ffa8748d05b6cd80" gracePeriod=30
Jul 28 13:17:56.820741 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:56.820498    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kube-rbac-proxy" containerID="cri-o://5101f3eb61151b27332f7c7b041bf88589b338aba17347c5646503a891077052" gracePeriod=30
Jul 28 13:17:57.546565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.546515    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 13:17:57.551949 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.551913    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-lightgbm-v2-runtime/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:17:57.735469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.735438    2576 generic.go:356] "Generic (PLEG): container finished" podID="383af136-d253-43a6-9d8b-f2548964f684" containerID="5101f3eb61151b27332f7c7b041bf88589b338aba17347c5646503a891077052" exitCode=2
Jul 28 13:17:57.735469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.735463    2576 generic.go:356] "Generic (PLEG): container finished" podID="383af136-d253-43a6-9d8b-f2548964f684" containerID="a307cce0a8a75b798a0be562d07b96ee45105608e83a5a08ffa8748d05b6cd80" exitCode=0
Jul 28 13:17:57.735656 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.735515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerDied","Data":"5101f3eb61151b27332f7c7b041bf88589b338aba17347c5646503a891077052"}
Jul 28 13:17:57.735656 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.735563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerDied","Data":"a307cce0a8a75b798a0be562d07b96ee45105608e83a5a08ffa8748d05b6cd80"}
Jul 28 13:17:57.766656 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.766634    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:17:57.835509 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835490    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383af136-d253-43a6-9d8b-f2548964f684-kserve-provision-location\") pod \"383af136-d253-43a6-9d8b-f2548964f684\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") "
Jul 28 13:17:57.835913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835537    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383af136-d253-43a6-9d8b-f2548964f684-proxy-tls\") pod \"383af136-d253-43a6-9d8b-f2548964f684\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") "
Jul 28 13:17:57.835913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835555    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-kube-rbac-proxy-sar-config\") pod \"383af136-d253-43a6-9d8b-f2548964f684\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") "
Jul 28 13:17:57.835913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835689    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67m4q\" (UniqueName: \"kubernetes.io/projected/383af136-d253-43a6-9d8b-f2548964f684-kube-api-access-67m4q\") pod \"383af136-d253-43a6-9d8b-f2548964f684\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") "
Jul 28 13:17:57.835913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835758    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-cabundle-cert\") pod \"383af136-d253-43a6-9d8b-f2548964f684\" (UID: \"383af136-d253-43a6-9d8b-f2548964f684\") "
Jul 28 13:17:57.835913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835898    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "383af136-d253-43a6-9d8b-f2548964f684" (UID: "383af136-d253-43a6-9d8b-f2548964f684"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:17:57.835913 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.835898    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383af136-d253-43a6-9d8b-f2548964f684-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "383af136-d253-43a6-9d8b-f2548964f684" (UID: "383af136-d253-43a6-9d8b-f2548964f684"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:17:57.836149 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.836110    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/383af136-d253-43a6-9d8b-f2548964f684-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:17:57.836149 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.836115    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "383af136-d253-43a6-9d8b-f2548964f684" (UID: "383af136-d253-43a6-9d8b-f2548964f684"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:17:57.836149 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.836128    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:17:57.837557 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.837537    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383af136-d253-43a6-9d8b-f2548964f684-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "383af136-d253-43a6-9d8b-f2548964f684" (UID: "383af136-d253-43a6-9d8b-f2548964f684"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:17:57.837666 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.837645    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383af136-d253-43a6-9d8b-f2548964f684-kube-api-access-67m4q" (OuterVolumeSpecName: "kube-api-access-67m4q") pod "383af136-d253-43a6-9d8b-f2548964f684" (UID: "383af136-d253-43a6-9d8b-f2548964f684"). InnerVolumeSpecName "kube-api-access-67m4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:17:57.937325 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.937301    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/383af136-d253-43a6-9d8b-f2548964f684-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:17:57.937325 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.937324    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67m4q\" (UniqueName: \"kubernetes.io/projected/383af136-d253-43a6-9d8b-f2548964f684-kube-api-access-67m4q\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:17:57.937325 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:57.937334    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/383af136-d253-43a6-9d8b-f2548964f684-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:17:58.741806 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.741770    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"
Jul 28 13:17:58.741806 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.741781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp" event={"ID":"383af136-d253-43a6-9d8b-f2548964f684","Type":"ContainerDied","Data":"cc083bc0d09c58ce55ab00b09097cbd8117944ea9bc9c9b08619dd48c7c7d841"}
Jul 28 13:17:58.742044 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.741847    2576 scope.go:117] "RemoveContainer" containerID="5101f3eb61151b27332f7c7b041bf88589b338aba17347c5646503a891077052"
Jul 28 13:17:58.750516 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.750497    2576 scope.go:117] "RemoveContainer" containerID="a307cce0a8a75b798a0be562d07b96ee45105608e83a5a08ffa8748d05b6cd80"
Jul 28 13:17:58.757940 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.757921    2576 scope.go:117] "RemoveContainer" containerID="e98217477af27a33e2fa93d27c98b7a2e78e5be964403dda6aa4f46d4d4f3c42"
Jul 28 13:17:58.766223 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.766173    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"]
Jul 28 13:17:58.768577 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:58.768558    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-txxmp"]
Jul 28 13:17:59.719711 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:17:59.719681    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383af136-d253-43a6-9d8b-f2548964f684" path="/var/lib/kubelet/pods/383af136-d253-43a6-9d8b-f2548964f684/volumes"
Jul 28 13:18:27.184970 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.184934    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"]
Jul 28 13:18:27.185578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185484    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kube-rbac-proxy"
Jul 28 13:18:27.185578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185502    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kube-rbac-proxy"
Jul 28 13:18:27.185578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185517    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kserve-container"
Jul 28 13:18:27.185578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185528    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kserve-container"
Jul 28 13:18:27.185578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185543    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="storage-initializer"
Jul 28 13:18:27.185578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185552    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="storage-initializer"
Jul 28 13:18:27.185925 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185633    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kube-rbac-proxy"
Jul 28 13:18:27.185925 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.185646    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="383af136-d253-43a6-9d8b-f2548964f684" containerName="kserve-container"
Jul 28 13:18:27.187765 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.187744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.190589 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.190565    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:18:27.190728 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.190595    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:18:27.190728 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.190672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:18:27.190856 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.190799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 13:18:27.190922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.190895    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:18:27.191879 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.191859    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:18:27.191985 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.191902    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:18:27.199575 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.199553    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"]
Jul 28 13:18:27.268932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.268907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.269051 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.268959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924cl\" (UniqueName: \"kubernetes.io/projected/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-api-access-924cl\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.269051 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.269019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19bf003b-cf32-485d-881d-856f5a50e3cd-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.269051 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.269037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19bf003b-cf32-485d-881d-856f5a50e3cd-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.269152 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.269055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.370264 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.370237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19bf003b-cf32-485d-881d-856f5a50e3cd-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.370398 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.370268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19bf003b-cf32-485d-881d-856f5a50e3cd-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.370398 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.370289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.370398 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.370324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.370398 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.370391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924cl\" (UniqueName: \"kubernetes.io/projected/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-api-access-924cl\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.370825 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.370802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19bf003b-cf32-485d-881d-856f5a50e3cd-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.371111 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.371094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.371163 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.371093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.372873 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.372852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19bf003b-cf32-485d-881d-856f5a50e3cd-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.379069 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.379051    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-924cl\" (UniqueName: \"kubernetes.io/projected/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-api-access-924cl\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.499587 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.499521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:27.627163 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.627138    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"]
Jul 28 13:18:27.629060 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:18:27.629035    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bf003b_cf32_485d_881d_856f5a50e3cd.slice/crio-b689530d62280337bbb97263eb467ee486319622da872d2c8f396cb54ac22036 WatchSource:0}: Error finding container b689530d62280337bbb97263eb467ee486319622da872d2c8f396cb54ac22036: Status 404 returned error can't find the container with id b689530d62280337bbb97263eb467ee486319622da872d2c8f396cb54ac22036
Jul 28 13:18:27.862352 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.862319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerStarted","Data":"ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d"}
Jul 28 13:18:27.862514 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:27.862392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerStarted","Data":"b689530d62280337bbb97263eb467ee486319622da872d2c8f396cb54ac22036"}
Jul 28 13:18:32.884421 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:32.884390    2576 generic.go:356] "Generic (PLEG): container finished" podID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerID="ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d" exitCode=0
Jul 28 13:18:32.884874 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:32.884439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerDied","Data":"ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d"}
Jul 28 13:18:33.890703 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:33.890673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerStarted","Data":"802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253"}
Jul 28 13:18:33.891168 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:33.890742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerStarted","Data":"f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4"}
Jul 28 13:18:33.891168 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:33.890975    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:33.924178 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:33.920543    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" podStartSLOduration=2.692873175 podStartE2EDuration="6.920524999s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.227651824s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:18:33.919302018 +0000 UTC m=+2606.820446421" watchObservedRunningTime="2026-07-28 13:18:33.920524999 +0000 UTC m=+2606.821669390"
Jul 28 13:18:34.896389 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:34.896354    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:18:40.905736 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:18:40.905707    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:19:10.909753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:10.909724    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:19:17.379870 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:17.379838    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"]
Jul 28 13:19:17.380387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:17.380073    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kserve-container" containerID="cri-o://f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4" gracePeriod=30
Jul 28 13:19:17.380387 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:17.380104    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kube-rbac-proxy" containerID="cri-o://802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253" gracePeriod=30
Jul 28 13:19:18.077032 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.076989    2576 generic.go:356] "Generic (PLEG): container finished" podID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerID="802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253" exitCode=2
Jul 28 13:19:18.077210 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.077040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerDied","Data":"802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253"}
Jul 28 13:19:18.521436 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.521414    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:19:18.566921 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.566897    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19bf003b-cf32-485d-881d-856f5a50e3cd-proxy-tls\") pod \"19bf003b-cf32-485d-881d-856f5a50e3cd\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") "
Jul 28 13:19:18.567060 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.566931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-924cl\" (UniqueName: \"kubernetes.io/projected/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-api-access-924cl\") pod \"19bf003b-cf32-485d-881d-856f5a50e3cd\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") "
Jul 28 13:19:18.567126 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.567074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-rbac-proxy-sar-config\") pod \"19bf003b-cf32-485d-881d-856f5a50e3cd\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") "
Jul 28 13:19:18.567190 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.567132    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19bf003b-cf32-485d-881d-856f5a50e3cd-kserve-provision-location\") pod \"19bf003b-cf32-485d-881d-856f5a50e3cd\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") "
Jul 28 13:19:18.567190 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.567169    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-cabundle-cert\") pod \"19bf003b-cf32-485d-881d-856f5a50e3cd\" (UID: \"19bf003b-cf32-485d-881d-856f5a50e3cd\") "
Jul 28 13:19:18.567448 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.567423    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19bf003b-cf32-485d-881d-856f5a50e3cd" (UID: "19bf003b-cf32-485d-881d-856f5a50e3cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:19:18.567531 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.567499    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bf003b-cf32-485d-881d-856f5a50e3cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19bf003b-cf32-485d-881d-856f5a50e3cd" (UID: "19bf003b-cf32-485d-881d-856f5a50e3cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:19:18.567632 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.567610    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19bf003b-cf32-485d-881d-856f5a50e3cd" (UID: "19bf003b-cf32-485d-881d-856f5a50e3cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:19:18.569002 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.568974    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-api-access-924cl" (OuterVolumeSpecName: "kube-api-access-924cl") pod "19bf003b-cf32-485d-881d-856f5a50e3cd" (UID: "19bf003b-cf32-485d-881d-856f5a50e3cd"). InnerVolumeSpecName "kube-api-access-924cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:19:18.569002 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.568992    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bf003b-cf32-485d-881d-856f5a50e3cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19bf003b-cf32-485d-881d-856f5a50e3cd" (UID: "19bf003b-cf32-485d-881d-856f5a50e3cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:19:18.668565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.668492    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:19:18.668565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.668527    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19bf003b-cf32-485d-881d-856f5a50e3cd-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:19:18.668565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.668537    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19bf003b-cf32-485d-881d-856f5a50e3cd-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:19:18.668565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.668546    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19bf003b-cf32-485d-881d-856f5a50e3cd-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:19:18.668565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:18.668554    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-924cl\" (UniqueName: \"kubernetes.io/projected/19bf003b-cf32-485d-881d-856f5a50e3cd-kube-api-access-924cl\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:19:19.083523 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.083495    2576 generic.go:356] "Generic (PLEG): container finished" podID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerID="f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4" exitCode=0
Jul 28 13:19:19.083680 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.083559    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"
Jul 28 13:19:19.083680 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.083561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerDied","Data":"f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4"}
Jul 28 13:19:19.083680 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.083635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh" event={"ID":"19bf003b-cf32-485d-881d-856f5a50e3cd","Type":"ContainerDied","Data":"b689530d62280337bbb97263eb467ee486319622da872d2c8f396cb54ac22036"}
Jul 28 13:19:19.083680 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.083661    2576 scope.go:117] "RemoveContainer" containerID="802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253"
Jul 28 13:19:19.092455 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.092440    2576 scope.go:117] "RemoveContainer" containerID="f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4"
Jul 28 13:19:19.099900 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.099879    2576 scope.go:117] "RemoveContainer" containerID="ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d"
Jul 28 13:19:19.106837 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.106810    2576 scope.go:117] "RemoveContainer" containerID="802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253"
Jul 28 13:19:19.107096 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:19:19.107076    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253\": container with ID starting with 802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253 not found: ID does not exist" containerID="802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253"
Jul 28 13:19:19.107139 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.107105    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253"} err="failed to get container status \"802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253\": rpc error: code = NotFound desc = could not find container \"802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253\": container with ID starting with 802c55b01f6e28224105a52e2f214ec06a3567df11906a560a25b01f0cfb1253 not found: ID does not exist"
Jul 28 13:19:19.107139 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.107127    2576 scope.go:117] "RemoveContainer" containerID="f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4"
Jul 28 13:19:19.107422 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:19:19.107401    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4\": container with ID starting with f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4 not found: ID does not exist" containerID="f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4"
Jul 28 13:19:19.107470 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.107429    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4"} err="failed to get container status \"f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4\": rpc error: code = NotFound desc = could not find container \"f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4\": container with ID starting with f93993af942832d7927411585189169220e9c76a326b1ef230b7def2be717ab4 not found: ID does not exist"
Jul 28 13:19:19.107470 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.107445    2576 scope.go:117] "RemoveContainer" containerID="ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d"
Jul 28 13:19:19.107684 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:19:19.107667    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d\": container with ID starting with ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d not found: ID does not exist" containerID="ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d"
Jul 28 13:19:19.107727 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.107689    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d"} err="failed to get container status \"ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d\": rpc error: code = NotFound desc = could not find container \"ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d\": container with ID starting with ccf3208c4700781e52e8a6c7228516675e932464d24eb8a8102e18b22826da5d not found: ID does not exist"
Jul 28 13:19:19.119595 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.119575    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"]
Jul 28 13:19:19.126620 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.126596    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-bdtfh"]
Jul 28 13:19:19.719891 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:19.719857    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" path="/var/lib/kubelet/pods/19bf003b-cf32-485d-881d-856f5a50e3cd/volumes"
Jul 28 13:19:22.481737 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.481702    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"]
Jul 28 13:19:22.482209 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482157    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kserve-container"
Jul 28 13:19:22.482209 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482171    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kserve-container"
Jul 28 13:19:22.482209 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482188    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kube-rbac-proxy"
Jul 28 13:19:22.482209 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482197    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kube-rbac-proxy"
Jul 28 13:19:22.482360 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482213    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="storage-initializer"
Jul 28 13:19:22.482360 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482222    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="storage-initializer"
Jul 28 13:19:22.482360 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482283    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kserve-container"
Jul 28 13:19:22.482360 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.482297    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bf003b-cf32-485d-881d-856f5a50e3cd" containerName="kube-rbac-proxy"
Jul 28 13:19:22.487032 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.487012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.489677 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.489652    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 13:19:22.489827 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.489800    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:19:22.489991 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.489971    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:19:22.489991 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.489985    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 13:19:22.490224 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.490210    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:19:22.490302 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.490279    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:19:22.490950 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.490935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:19:22.496438 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.496413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"]
Jul 28 13:19:22.602588 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.602561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.602756 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.602591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpwz\" (UniqueName: \"kubernetes.io/projected/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-api-access-5cpwz\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.602756 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.602628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.602756 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.602649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.602756 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.602738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.703841 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.703810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.703841 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.703843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.704064 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.703886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.704064 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.703918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.704064 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.703934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpwz\" (UniqueName: \"kubernetes.io/projected/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-api-access-5cpwz\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.704183 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:19:22.704076    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 13:19:22.704183 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:19:22.704152    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls podName:6c87168a-8e45-4d4d-9ba3-fb2ce15dc986 nodeName:}" failed. No retries permitted until 2026-07-28 13:19:23.204130746 +0000 UTC m=+2656.105275121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls") pod "isvc-sklearn-mcp-predictor-dc795d948-prxrc" (UID: "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 13:19:22.704411 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.704391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.704519 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.704501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.704573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.704559    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:22.715852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:22.715829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpwz\" (UniqueName: \"kubernetes.io/projected/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-api-access-5cpwz\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:23.208655 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:23.208622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:23.210965 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:23.210941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-dc795d948-prxrc\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:23.398465 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:23.398436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:23.530328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:23.530300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"]
Jul 28 13:19:23.533448 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:19:23.533420    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c87168a_8e45_4d4d_9ba3_fb2ce15dc986.slice/crio-c6833a65bee5062aebeb5d937f9d1f26cbff384188d0ef6b801a1252a297eab3 WatchSource:0}: Error finding container c6833a65bee5062aebeb5d937f9d1f26cbff384188d0ef6b801a1252a297eab3: Status 404 returned error can't find the container with id c6833a65bee5062aebeb5d937f9d1f26cbff384188d0ef6b801a1252a297eab3
Jul 28 13:19:24.106662 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:24.106621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerStarted","Data":"b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e"}
Jul 28 13:19:24.106842 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:24.106686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerStarted","Data":"c6833a65bee5062aebeb5d937f9d1f26cbff384188d0ef6b801a1252a297eab3"}
Jul 28 13:19:28.130925 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:28.130890    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerID="b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e" exitCode=0
Jul 28 13:19:28.131471 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:28.130961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerDied","Data":"b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e"}
Jul 28 13:19:28.196541 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:28.196524    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:19:29.139229 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:29.139194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerStarted","Data":"054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad"}
Jul 28 13:19:31.151901 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:31.151864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerStarted","Data":"f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f"}
Jul 28 13:19:31.152291 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:31.151930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerStarted","Data":"62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a"}
Jul 28 13:19:31.152291 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:31.151980    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:31.152291 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:31.152040    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:31.152291 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:31.152066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:19:31.178300 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:31.178215    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podStartSLOduration=2.596442109 podStartE2EDuration="9.178201138s" totalImagesPullingTime="2.727228641s" totalInitContainerRuntime="3.854530388s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:19:31.176555131 +0000 UTC m=+2664.077699523" watchObservedRunningTime="2026-07-28 13:19:31.178201138 +0000 UTC m=+2664.079345528"
Jul 28 13:19:37.168330 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:19:37.168296    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:20:03.399950 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:03.399906    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:20:07.172933 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:07.172902    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:20:37.174202 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:37.174168    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:20:42.597563 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:42.597531    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"]
Jul 28 13:20:42.598024 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:42.597848    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container" containerID="cri-o://054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad" gracePeriod=30
Jul 28 13:20:42.598024 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:42.597866    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" containerID="cri-o://f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f" gracePeriod=30
Jul 28 13:20:42.598024 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:42.597873    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-agent" containerID="cri-o://62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a" gracePeriod=30
Jul 28 13:20:43.480727 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:43.480686    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerID="f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f" exitCode=2
Jul 28 13:20:43.480907 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:43.480762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerDied","Data":"f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f"}
Jul 28 13:20:45.492776 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:45.492744    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerID="054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad" exitCode=0
Jul 28 13:20:45.493141 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:45.492795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerDied","Data":"054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad"}
Jul 28 13:20:47.163768 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:47.163728    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 13:20:47.169132 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:47.169105    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:20:52.163464 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:52.163419    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 13:20:57.163825 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:57.163786    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 13:20:57.164268 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:57.163925    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:20:57.169148 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:20:57.169115    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:21:02.163286 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:02.163235    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 13:21:07.163862 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:07.163822    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 13:21:07.169270 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:07.169241    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:21:07.169364 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:07.169353    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:21:12.163175 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.163127    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 13:21:12.770672 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.770647    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:21:12.900868 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.900796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kserve-provision-location\") pod \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") "
Jul 28 13:21:12.901008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.900870    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-rbac-proxy-sar-config\") pod \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") "
Jul 28 13:21:12.901008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.900895    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cpwz\" (UniqueName: \"kubernetes.io/projected/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-api-access-5cpwz\") pod \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") "
Jul 28 13:21:12.901008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.900922    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls\") pod \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") "
Jul 28 13:21:12.901008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.900940    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-cabundle-cert\") pod \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\" (UID: \"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986\") "
Jul 28 13:21:12.901221 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.901146    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" (UID: "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:21:12.901303 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.901277    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" (UID: "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:21:12.901429 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.901407    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" (UID: "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:21:12.902996 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.902972    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-api-access-5cpwz" (OuterVolumeSpecName: "kube-api-access-5cpwz") pod "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" (UID: "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986"). InnerVolumeSpecName "kube-api-access-5cpwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:21:12.903049 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:12.902990    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" (UID: "6c87168a-8e45-4d4d-9ba3-fb2ce15dc986"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:21:13.002057 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.002032    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:21:13.002057 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.002054    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpwz\" (UniqueName: \"kubernetes.io/projected/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kube-api-access-5cpwz\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:21:13.002214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.002064    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:21:13.002214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.002073    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:21:13.002214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.002081    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:21:13.611292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.611259    2576 generic.go:356] "Generic (PLEG): container finished" podID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerID="62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a" exitCode=0
Jul 28 13:21:13.611667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.611316    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"
Jul 28 13:21:13.611667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.611346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerDied","Data":"62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a"}
Jul 28 13:21:13.611667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.611428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc" event={"ID":"6c87168a-8e45-4d4d-9ba3-fb2ce15dc986","Type":"ContainerDied","Data":"c6833a65bee5062aebeb5d937f9d1f26cbff384188d0ef6b801a1252a297eab3"}
Jul 28 13:21:13.611667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.611462    2576 scope.go:117] "RemoveContainer" containerID="f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f"
Jul 28 13:21:13.621183 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.621169    2576 scope.go:117] "RemoveContainer" containerID="62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a"
Jul 28 13:21:13.628415 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.628392    2576 scope.go:117] "RemoveContainer" containerID="054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad"
Jul 28 13:21:13.635736 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.635712    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"]
Jul 28 13:21:13.636472 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.636455    2576 scope.go:117] "RemoveContainer" containerID="b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e"
Jul 28 13:21:13.643949 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.643927    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-dc795d948-prxrc"]
Jul 28 13:21:13.644968 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.644949    2576 scope.go:117] "RemoveContainer" containerID="f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f"
Jul 28 13:21:13.645211 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:21:13.645191    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f\": container with ID starting with f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f not found: ID does not exist" containerID="f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f"
Jul 28 13:21:13.645279 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645221    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f"} err="failed to get container status \"f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f\": rpc error: code = NotFound desc = could not find container \"f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f\": container with ID starting with f07aaafc28837a8dd86e99fec68278935b34e733ae55eefc04ee60c84565e64f not found: ID does not exist"
Jul 28 13:21:13.645279 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645241    2576 scope.go:117] "RemoveContainer" containerID="62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a"
Jul 28 13:21:13.645517 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:21:13.645500    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a\": container with ID starting with 62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a not found: ID does not exist" containerID="62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a"
Jul 28 13:21:13.645574 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645523    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a"} err="failed to get container status \"62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a\": rpc error: code = NotFound desc = could not find container \"62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a\": container with ID starting with 62115948f1cd88b1eeb5f3f351d2bd9d4f4a4508679f6ec20456791b12403a3a not found: ID does not exist"
Jul 28 13:21:13.645574 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645538    2576 scope.go:117] "RemoveContainer" containerID="054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad"
Jul 28 13:21:13.645759 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:21:13.645742    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad\": container with ID starting with 054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad not found: ID does not exist" containerID="054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad"
Jul 28 13:21:13.645796 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645764    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad"} err="failed to get container status \"054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad\": rpc error: code = NotFound desc = could not find container \"054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad\": container with ID starting with 054a17ec206fd6d8efeb460ba523a2e86f464f1cebed938906456f58b72bc2ad not found: ID does not exist"
Jul 28 13:21:13.645796 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645781    2576 scope.go:117] "RemoveContainer" containerID="b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e"
Jul 28 13:21:13.645983 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:21:13.645966    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e\": container with ID starting with b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e not found: ID does not exist" containerID="b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e"
Jul 28 13:21:13.646031 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.645986    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e"} err="failed to get container status \"b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e\": rpc error: code = NotFound desc = could not find container \"b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e\": container with ID starting with b7d3454c23ba02d7b10deed8134f5c43b4bf4bd100f415a65cc4bc1b7fdc780e not found: ID does not exist"
Jul 28 13:21:13.719891 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:21:13.719865    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" path="/var/lib/kubelet/pods/6c87168a-8e45-4d4d-9ba3-fb2ce15dc986/volumes"
Jul 28 13:23:11.217563 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217526    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"]
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217883    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217895    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217906    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="storage-initializer"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217911    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="storage-initializer"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217924    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-agent"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217929    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-agent"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217942    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container"
Jul 28 13:23:11.218021 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.217951    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container"
Jul 28 13:23:11.218277 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.218031    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kube-rbac-proxy"
Jul 28 13:23:11.218277 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.218039    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-agent"
Jul 28 13:23:11.218277 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.218046    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c87168a-8e45-4d4d-9ba3-fb2ce15dc986" containerName="kserve-container"
Jul 28 13:23:11.220166 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.220147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.223252 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.223220    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:23:11.223405 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.223269    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:23:11.223405 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.223306    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 13:23:11.223405 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.223219    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:23:11.224258 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.224240    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 13:23:11.224341 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.224311    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:23:11.224421 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.224344    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:23:11.237894 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.237871    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"]
Jul 28 13:23:11.291976 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.291947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmjn\" (UniqueName: \"kubernetes.io/projected/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-api-access-bqmjn\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.292135 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.291989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.292135 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.292015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4816c60-955d-4bf3-b0c0-f0572d4ea789-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.292135 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.292104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.292283 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.292140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.392932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.392906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmjn\" (UniqueName: \"kubernetes.io/projected/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-api-access-bqmjn\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.392961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.392995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4816c60-955d-4bf3-b0c0-f0572d4ea789-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.393041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393105 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.393067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393521 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.393503    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393695 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.393666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.393828 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.393776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.395705 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.395688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4816c60-955d-4bf3-b0c0-f0572d4ea789-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.401167 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.401145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmjn\" (UniqueName: \"kubernetes.io/projected/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-api-access-bqmjn\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.530802 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.530727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:11.660091 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:11.660056    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"]
Jul 28 13:23:11.663073 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:23:11.663045    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4816c60_955d_4bf3_b0c0_f0572d4ea789.slice/crio-f4e3fb07e079bcb1809bd995da0011ad80c0ba7edbf3e1030d028351c0e5ce2b WatchSource:0}: Error finding container f4e3fb07e079bcb1809bd995da0011ad80c0ba7edbf3e1030d028351c0e5ce2b: Status 404 returned error can't find the container with id f4e3fb07e079bcb1809bd995da0011ad80c0ba7edbf3e1030d028351c0e5ce2b
Jul 28 13:23:12.108101 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:12.108052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerStarted","Data":"f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb"}
Jul 28 13:23:12.108272 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:12.108133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerStarted","Data":"f4e3fb07e079bcb1809bd995da0011ad80c0ba7edbf3e1030d028351c0e5ce2b"}
Jul 28 13:23:16.126738 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:16.126711    2576 generic.go:356] "Generic (PLEG): container finished" podID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerID="f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb" exitCode=0
Jul 28 13:23:16.127097 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:16.126788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerDied","Data":"f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb"}
Jul 28 13:23:28.191640 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:28.191614    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerStarted","Data":"93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8"}
Jul 28 13:23:29.197190 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:29.197156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerStarted","Data":"2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4"}
Jul 28 13:23:29.197616 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:29.197216    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:29.197616 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:29.197267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:29.198543 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:29.198520    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:23:29.219488 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:29.219438    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podStartSLOduration=1.992644636 podStartE2EDuration="18.219424111s" totalImagesPullingTime="11.956324525s" totalInitContainerRuntime="4.27045495s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:23:29.219026989 +0000 UTC m=+2902.120171378" watchObservedRunningTime="2026-07-28 13:23:29.219424111 +0000 UTC m=+2902.120568502"
Jul 28 13:23:30.202691 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:30.202657    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:23:35.207434 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:35.207405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:23:35.207949 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:35.207914    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:23:45.208090 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:45.208043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:23:55.208821 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:23:55.208776    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:24:05.208527 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:05.208486    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:24:15.208551 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:15.208518    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:24:23.092315 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:23.092283    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"]
Jul 28 13:24:23.092888 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:23.092564    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" containerID="cri-o://93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8" gracePeriod=30
Jul 28 13:24:23.092888 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:23.092595    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kube-rbac-proxy" containerID="cri-o://2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4" gracePeriod=30
Jul 28 13:24:23.435199 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:23.435119    2576 generic.go:356] "Generic (PLEG): container finished" podID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerID="2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4" exitCode=2
Jul 28 13:24:23.435324 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:23.435193    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerDied","Data":"2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4"}
Jul 28 13:24:25.203937 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.203891    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 28 13:24:25.208307 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.208276    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 13:24:25.646068 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.646042    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:24:25.803293 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqmjn\" (UniqueName: \"kubernetes.io/projected/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-api-access-bqmjn\") pod \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") "
Jul 28 13:24:25.803293 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803274    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-cabundle-cert\") pod \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") "
Jul 28 13:24:25.803519 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803311    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-rbac-proxy-sar-config\") pod \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") "
Jul 28 13:24:25.803519 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803332    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kserve-provision-location\") pod \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") "
Jul 28 13:24:25.803519 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4816c60-955d-4bf3-b0c0-f0572d4ea789-proxy-tls\") pod \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\" (UID: \"a4816c60-955d-4bf3-b0c0-f0572d4ea789\") "
Jul 28 13:24:25.803766 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4816c60-955d-4bf3-b0c0-f0572d4ea789" (UID: "a4816c60-955d-4bf3-b0c0-f0572d4ea789"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:24:25.803838 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.803737    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4816c60-955d-4bf3-b0c0-f0572d4ea789" (UID: "a4816c60-955d-4bf3-b0c0-f0572d4ea789"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:24:25.805297 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.805271    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-api-access-bqmjn" (OuterVolumeSpecName: "kube-api-access-bqmjn") pod "a4816c60-955d-4bf3-b0c0-f0572d4ea789" (UID: "a4816c60-955d-4bf3-b0c0-f0572d4ea789"). InnerVolumeSpecName "kube-api-access-bqmjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:24:25.805416 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.805317    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4816c60-955d-4bf3-b0c0-f0572d4ea789-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4816c60-955d-4bf3-b0c0-f0572d4ea789" (UID: "a4816c60-955d-4bf3-b0c0-f0572d4ea789"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:24:25.813493 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.813470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4816c60-955d-4bf3-b0c0-f0572d4ea789" (UID: "a4816c60-955d-4bf3-b0c0-f0572d4ea789"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:24:25.904618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.904596    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:24:25.904618 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.904617    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:24:25.904764 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.904627    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4816c60-955d-4bf3-b0c0-f0572d4ea789-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:24:25.904764 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.904637    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmjn\" (UniqueName: \"kubernetes.io/projected/a4816c60-955d-4bf3-b0c0-f0572d4ea789-kube-api-access-bqmjn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:24:25.904764 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:25.904646    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4816c60-955d-4bf3-b0c0-f0572d4ea789-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:24:26.448476 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.448441    2576 generic.go:356] "Generic (PLEG): container finished" podID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerID="93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8" exitCode=0
Jul 28 13:24:26.448843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.448494    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"
Jul 28 13:24:26.448843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.448522    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerDied","Data":"93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8"}
Jul 28 13:24:26.448843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.448585    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb" event={"ID":"a4816c60-955d-4bf3-b0c0-f0572d4ea789","Type":"ContainerDied","Data":"f4e3fb07e079bcb1809bd995da0011ad80c0ba7edbf3e1030d028351c0e5ce2b"}
Jul 28 13:24:26.448843 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.448611    2576 scope.go:117] "RemoveContainer" containerID="2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4"
Jul 28 13:24:26.457178 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.457159    2576 scope.go:117] "RemoveContainer" containerID="93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8"
Jul 28 13:24:26.464533 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.464514    2576 scope.go:117] "RemoveContainer" containerID="f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb"
Jul 28 13:24:26.471722 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.471699    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"]
Jul 28 13:24:26.472242 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.472224    2576 scope.go:117] "RemoveContainer" containerID="2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4"
Jul 28 13:24:26.472497 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:24:26.472481    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4\": container with ID starting with 2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4 not found: ID does not exist" containerID="2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4"
Jul 28 13:24:26.472552 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.472504    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4"} err="failed to get container status \"2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4\": rpc error: code = NotFound desc = could not find container \"2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4\": container with ID starting with 2ca0cd1ba7d59e20cbb3b3c94467452791ba654d39c406f752c4016f4b3566d4 not found: ID does not exist"
Jul 28 13:24:26.472552 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.472523    2576 scope.go:117] "RemoveContainer" containerID="93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8"
Jul 28 13:24:26.472790 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:24:26.472771    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8\": container with ID starting with 93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8 not found: ID does not exist" containerID="93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8"
Jul 28 13:24:26.472848 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.472800    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8"} err="failed to get container status \"93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8\": rpc error: code = NotFound desc = could not find container \"93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8\": container with ID starting with 93bee97f47edce6abbedbd350879276934a09e5e5bde2688e7fd68fda84837e8 not found: ID does not exist"
Jul 28 13:24:26.472848 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.472823    2576 scope.go:117] "RemoveContainer" containerID="f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb"
Jul 28 13:24:26.473045 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:24:26.473031    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb\": container with ID starting with f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb not found: ID does not exist" containerID="f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb"
Jul 28 13:24:26.473096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.473052    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb"} err="failed to get container status \"f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb\": rpc error: code = NotFound desc = could not find container \"f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb\": container with ID starting with f19cb607c9f05220c0890fca5844c60d849a864ca50707ae167d021931c3d8eb not found: ID does not exist"
Jul 28 13:24:26.478768 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:26.478745    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-mrbgb"]
Jul 28 13:24:27.719565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:24:27.719530    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" path="/var/lib/kubelet/pods/a4816c60-955d-4bf3-b0c0-f0572d4ea789/volumes"
Jul 28 13:29:26.790269 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790236    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"]
Jul 28 13:29:26.790753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790700    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container"
Jul 28 13:29:26.790753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790716    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container"
Jul 28 13:29:26.790753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790732    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="storage-initializer"
Jul 28 13:29:26.790753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790738    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="storage-initializer"
Jul 28 13:29:26.790753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790752    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kube-rbac-proxy"
Jul 28 13:29:26.790753 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790758    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kube-rbac-proxy"
Jul 28 13:29:26.791096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790811    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kube-rbac-proxy"
Jul 28 13:29:26.791096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.790820    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4816c60-955d-4bf3-b0c0-f0572d4ea789" containerName="kserve-container"
Jul 28 13:29:26.794010 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.793987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.796615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.796591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a95d50-predictor-serving-cert\""
Jul 28 13:29:26.796851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.796701    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:29:26.796851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.796745    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:29:26.796851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.796793    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:29:26.797804 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.797775    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:29:26.797911 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.797814    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a95d50-kube-rbac-proxy-sar-config\""
Jul 28 13:29:26.797911 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.797819    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:29:26.803279 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.803256    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"]
Jul 28 13:29:26.809694 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.809667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvtv\" (UniqueName: \"kubernetes.io/projected/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-api-access-sqvtv\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.809784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.809698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-cabundle-cert\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.809835 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.809790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9050e79a-c9df-4d29-807c-7bab69ce1f08-kserve-provision-location\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.809960 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.809941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.810037 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.809988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.911295 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.911264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.911460 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.911327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvtv\" (UniqueName: \"kubernetes.io/projected/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-api-access-sqvtv\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.911460 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.911352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-cabundle-cert\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.911460 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.911421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9050e79a-c9df-4d29-807c-7bab69ce1f08-kserve-provision-location\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.911641 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.911544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.911698 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:29:26.911660    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-a95d50-predictor-serving-cert: secret "isvc-primary-a95d50-predictor-serving-cert" not found
Jul 28 13:29:26.911747 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:29:26.911741    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls podName:9050e79a-c9df-4d29-807c-7bab69ce1f08 nodeName:}" failed. No retries permitted until 2026-07-28 13:29:27.411719509 +0000 UTC m=+3260.312863877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls") pod "isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" (UID: "9050e79a-c9df-4d29-807c-7bab69ce1f08") : secret "isvc-primary-a95d50-predictor-serving-cert" not found
Jul 28 13:29:26.911820 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.911771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9050e79a-c9df-4d29-807c-7bab69ce1f08-kserve-provision-location\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.912027 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.912012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.912098 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.912084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-cabundle-cert\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:26.921860 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:26.921840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvtv\" (UniqueName: \"kubernetes.io/projected/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-api-access-sqvtv\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:27.415308 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:27.415284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:27.417524 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:27.417498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls\") pod \"isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") " pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:27.705181 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:27.705102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:27.837130 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:27.836976    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"]
Jul 28 13:29:27.839466 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:29:27.839439    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9050e79a_c9df_4d29_807c_7bab69ce1f08.slice/crio-ab6286293ea0e2d36faacf4f8315a653a5c6b04e70a5f30f0af05227366c990e WatchSource:0}: Error finding container ab6286293ea0e2d36faacf4f8315a653a5c6b04e70a5f30f0af05227366c990e: Status 404 returned error can't find the container with id ab6286293ea0e2d36faacf4f8315a653a5c6b04e70a5f30f0af05227366c990e
Jul 28 13:29:28.694276 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:28.694241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerStarted","Data":"fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536"}
Jul 28 13:29:28.694576 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:28.694307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerStarted","Data":"ab6286293ea0e2d36faacf4f8315a653a5c6b04e70a5f30f0af05227366c990e"}
Jul 28 13:29:31.709283 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:31.709252    2576 generic.go:356] "Generic (PLEG): container finished" podID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerID="fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536" exitCode=0
Jul 28 13:29:31.709700 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:31.709326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerDied","Data":"fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536"}
Jul 28 13:29:32.716292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:32.716260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerStarted","Data":"f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486"}
Jul 28 13:29:32.716671 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:32.716322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerStarted","Data":"2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6"}
Jul 28 13:29:32.716671 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:32.716514    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:32.738885 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:32.738836    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podStartSLOduration=3.119984475 podStartE2EDuration="6.738821601s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.618837126s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:29:32.736142733 +0000 UTC m=+3265.637287129" watchObservedRunningTime="2026-07-28 13:29:32.738821601 +0000 UTC m=+3265.639965990"
Jul 28 13:29:33.720316 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:33.720287    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:33.721717 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:33.721692    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:29:34.724960 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:34.724922    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:29:39.729563 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:39.729529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:29:39.730080 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:39.730054    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:29:49.730856 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:49.730814    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:29:59.730204 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:29:59.730165    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:30:09.730080 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:30:09.730035    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:30:19.730780 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:30:19.730741    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:30:29.730564 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:30:29.730482    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:30:39.730096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:30:39.730054    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 13:30:48.716256 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:30:48.716227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:31:30.034678 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:30.034647    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"]
Jul 28 13:31:30.035203 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:30.034948    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container" containerID="cri-o://2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6" gracePeriod=30
Jul 28 13:31:30.035203 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:30.034963    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kube-rbac-proxy" containerID="cri-o://f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486" gracePeriod=30
Jul 28 13:31:30.226202 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:30.226167    2576 generic.go:356] "Generic (PLEG): container finished" podID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerID="f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486" exitCode=2
Jul 28 13:31:30.226406 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:30.226236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerDied","Data":"f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486"}
Jul 28 13:31:34.476762 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.476733    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:31:34.605807 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.605777    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-rbac-proxy-sar-config\") pod \"9050e79a-c9df-4d29-807c-7bab69ce1f08\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") "
Jul 28 13:31:34.605965 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.605847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9050e79a-c9df-4d29-807c-7bab69ce1f08-kserve-provision-location\") pod \"9050e79a-c9df-4d29-807c-7bab69ce1f08\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") "
Jul 28 13:31:34.605965 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.605889    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqvtv\" (UniqueName: \"kubernetes.io/projected/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-api-access-sqvtv\") pod \"9050e79a-c9df-4d29-807c-7bab69ce1f08\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") "
Jul 28 13:31:34.605965 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.605950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-cabundle-cert\") pod \"9050e79a-c9df-4d29-807c-7bab69ce1f08\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") "
Jul 28 13:31:34.606093 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.605982    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls\") pod \"9050e79a-c9df-4d29-807c-7bab69ce1f08\" (UID: \"9050e79a-c9df-4d29-807c-7bab69ce1f08\") "
Jul 28 13:31:34.606214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.606186    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9050e79a-c9df-4d29-807c-7bab69ce1f08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9050e79a-c9df-4d29-807c-7bab69ce1f08" (UID: "9050e79a-c9df-4d29-807c-7bab69ce1f08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:31:34.606254 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.606191    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9050e79a-c9df-4d29-807c-7bab69ce1f08" (UID: "9050e79a-c9df-4d29-807c-7bab69ce1f08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:34.606303 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.606272    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:31:34.606303 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.606290    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9050e79a-c9df-4d29-807c-7bab69ce1f08-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:31:34.606303 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.606293    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9050e79a-c9df-4d29-807c-7bab69ce1f08" (UID: "9050e79a-c9df-4d29-807c-7bab69ce1f08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:31:34.608079 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.608048    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9050e79a-c9df-4d29-807c-7bab69ce1f08" (UID: "9050e79a-c9df-4d29-807c-7bab69ce1f08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:31:34.608196 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.608122    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-api-access-sqvtv" (OuterVolumeSpecName: "kube-api-access-sqvtv") pod "9050e79a-c9df-4d29-807c-7bab69ce1f08" (UID: "9050e79a-c9df-4d29-807c-7bab69ce1f08"). InnerVolumeSpecName "kube-api-access-sqvtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:31:34.707057 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.707031    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvtv\" (UniqueName: \"kubernetes.io/projected/9050e79a-c9df-4d29-807c-7bab69ce1f08-kube-api-access-sqvtv\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:31:34.707057 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.707059    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9050e79a-c9df-4d29-807c-7bab69ce1f08-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:31:34.707236 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:34.707071    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9050e79a-c9df-4d29-807c-7bab69ce1f08-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:31:35.249230 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.249197    2576 generic.go:356] "Generic (PLEG): container finished" podID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerID="2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6" exitCode=0
Jul 28 13:31:35.249230 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.249226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerDied","Data":"2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6"}
Jul 28 13:31:35.249443 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.249275    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"
Jul 28 13:31:35.249443 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.249288    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn" event={"ID":"9050e79a-c9df-4d29-807c-7bab69ce1f08","Type":"ContainerDied","Data":"ab6286293ea0e2d36faacf4f8315a653a5c6b04e70a5f30f0af05227366c990e"}
Jul 28 13:31:35.249443 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.249316    2576 scope.go:117] "RemoveContainer" containerID="f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486"
Jul 28 13:31:35.258311 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.258293    2576 scope.go:117] "RemoveContainer" containerID="2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6"
Jul 28 13:31:35.266217 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.266199    2576 scope.go:117] "RemoveContainer" containerID="fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536"
Jul 28 13:31:35.273939 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.273924    2576 scope.go:117] "RemoveContainer" containerID="f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486"
Jul 28 13:31:35.274212 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:31:35.274192    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486\": container with ID starting with f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486 not found: ID does not exist" containerID="f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486"
Jul 28 13:31:35.274270 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.274221    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486"} err="failed to get container status \"f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486\": rpc error: code = NotFound desc = could not find container \"f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486\": container with ID starting with f0abc6c5c567be3229a891c0fcc453e6d24b97c2a5f103a677c1070ad120b486 not found: ID does not exist"
Jul 28 13:31:35.274270 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.274241    2576 scope.go:117] "RemoveContainer" containerID="2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6"
Jul 28 13:31:35.274509 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:31:35.274490    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6\": container with ID starting with 2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6 not found: ID does not exist" containerID="2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6"
Jul 28 13:31:35.274562 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.274512    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6"} err="failed to get container status \"2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6\": rpc error: code = NotFound desc = could not find container \"2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6\": container with ID starting with 2fbc08f4a7b648802d18757129b295e4aaec5132e79b3d0ed035f63b996d7da6 not found: ID does not exist"
Jul 28 13:31:35.274562 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.274526    2576 scope.go:117] "RemoveContainer" containerID="fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536"
Jul 28 13:31:35.274763 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:31:35.274749    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536\": container with ID starting with fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536 not found: ID does not exist" containerID="fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536"
Jul 28 13:31:35.274805 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.274766    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536"} err="failed to get container status \"fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536\": rpc error: code = NotFound desc = could not find container \"fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536\": container with ID starting with fcc121f56610bac00e45055e453e550b9ab619c10970e2d064c1386ee0c77536 not found: ID does not exist"
Jul 28 13:31:35.277956 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.277934    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"]
Jul 28 13:31:35.286686 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.286663    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a95d50-predictor-84bb6b9cf7-fn7rn"]
Jul 28 13:31:35.719812 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:31:35.719781    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" path="/var/lib/kubelet/pods/9050e79a-c9df-4d29-807c-7bab69ce1f08/volumes"
Jul 28 13:33:35.875484 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875402    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"]
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875739    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kube-rbac-proxy"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875751    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kube-rbac-proxy"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875765    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="storage-initializer"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875771    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="storage-initializer"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875786    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875792    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875847    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kserve-container"
Jul 28 13:33:35.877988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.875853    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9050e79a-c9df-4d29-807c-7bab69ce1f08" containerName="kube-rbac-proxy"
Jul 28 13:33:35.878963 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.878948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:35.881768 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.881741    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 13:33:35.881898 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.881751    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 13:33:35.881898 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.881755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:33:35.882034 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.881990    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:33:35.882096 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.882042    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:33:35.882141 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.882097    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:33:35.882921 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.882903    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:33:35.890903 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.890884    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"]
Jul 28 13:33:35.967998 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.967963    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:35.967998 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.968000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:35.968222 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.968049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:35.968222 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.968069    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:35.968222 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:35.968088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lcwf\" (UniqueName: \"kubernetes.io/projected/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-api-access-7lcwf\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.068847 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.068810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.068847 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.068847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.069075 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.068887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.069075 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.068906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.069075 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.068925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lcwf\" (UniqueName: \"kubernetes.io/projected/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-api-access-7lcwf\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.069437 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.069411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.069636 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.069617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.069684 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.069620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.071419 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.071390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.077427 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.077406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lcwf\" (UniqueName: \"kubernetes.io/projected/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-api-access-7lcwf\") pod \"isvc-predictive-xgboost-predictor-695f978978-wskww\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.190239 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.190160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:33:36.320419 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.320391    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"]
Jul 28 13:33:36.322391 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:33:36.322341    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5ba7aa_5d4e_4e5f_9a67_8369d31e1fde.slice/crio-044f7e240dfaa1e869424e5791881da98e23efb51325ad404aab967809a54239 WatchSource:0}: Error finding container 044f7e240dfaa1e869424e5791881da98e23efb51325ad404aab967809a54239: Status 404 returned error can't find the container with id 044f7e240dfaa1e869424e5791881da98e23efb51325ad404aab967809a54239
Jul 28 13:33:36.756863 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.756823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerStarted","Data":"7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e"}
Jul 28 13:33:36.757143 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:36.756913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerStarted","Data":"044f7e240dfaa1e869424e5791881da98e23efb51325ad404aab967809a54239"}
Jul 28 13:33:40.776183 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:40.776150    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerID="7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e" exitCode=0
Jul 28 13:33:40.776651 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:40.776227    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerDied","Data":"7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e"}
Jul 28 13:33:40.777481 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:33:40.777467    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:34:01.898076 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:01.898043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerStarted","Data":"fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f"}
Jul 28 13:34:02.904921 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:02.904887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerStarted","Data":"f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396"}
Jul 28 13:34:02.905364 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:02.905161    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:34:02.905364 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:02.905199    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:34:02.906625 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:02.906599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:02.923947 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:02.923905    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podStartSLOduration=3.271941396 podStartE2EDuration="27.923892376s" totalImagesPullingTime="21.010660348s" totalInitContainerRuntime="3.641290632s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:34:02.922692866 +0000 UTC m=+3535.823837255" watchObservedRunningTime="2026-07-28 13:34:02.923892376 +0000 UTC m=+3535.825036819"
Jul 28 13:34:03.909622 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:03.909575    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:08.914333 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:08.914304    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:34:08.914941 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:08.914839    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:18.915116 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:18.915072    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:28.915321 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:28.915280    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:38.915654 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:38.915615    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:48.915345 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:48.915304    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:34:58.915132 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:34:58.915094    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:35:08.914815 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:08.914774    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:35:18.915565 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:18.915530    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:35:26.032062 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:26.032022    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"]
Jul 28 13:35:26.032558 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:26.032275    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" containerID="cri-o://fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f" gracePeriod=30
Jul 28 13:35:26.032558 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:26.032323    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kube-rbac-proxy" containerID="cri-o://f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396" gracePeriod=30
Jul 28 13:35:26.259806 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:26.259774    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerID="f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396" exitCode=2
Jul 28 13:35:26.259942 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:26.259850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerDied","Data":"f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396"}
Jul 28 13:35:28.909997 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:28.909958    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 28 13:35:28.915322 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:28.915290    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 13:35:30.568780 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.568746    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:35:30.632349 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632316    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-proxy-tls\") pod \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") "
Jul 28 13:35:30.632518 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-rbac-proxy-sar-config\") pod \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") "
Jul 28 13:35:30.632518 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kserve-provision-location\") pod \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") "
Jul 28 13:35:30.632518 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632463    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lcwf\" (UniqueName: \"kubernetes.io/projected/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-api-access-7lcwf\") pod \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") "
Jul 28 13:35:30.632518 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632493    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-cabundle-cert\") pod \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\" (UID: \"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde\") "
Jul 28 13:35:30.632760 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632731    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" (UID: "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:35:30.632848 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632732    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" (UID: "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:35:30.632968 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.632902    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" (UID: "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:35:30.634448 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.634429    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" (UID: "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:35:30.634448 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.634430    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-api-access-7lcwf" (OuterVolumeSpecName: "kube-api-access-7lcwf") pod "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" (UID: "ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde"). InnerVolumeSpecName "kube-api-access-7lcwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:35:30.733882 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.733823    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:35:30.733882 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.733844    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:35:30.733882 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.733854    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:35:30.733882 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.733864    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lcwf\" (UniqueName: \"kubernetes.io/projected/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-kube-api-access-7lcwf\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:35:30.733882 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:30.733872    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:35:31.286646 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.286614    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerID="fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f" exitCode=0
Jul 28 13:35:31.286823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.286668    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"
Jul 28 13:35:31.286823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.286684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerDied","Data":"fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f"}
Jul 28 13:35:31.286823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.286746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww" event={"ID":"ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde","Type":"ContainerDied","Data":"044f7e240dfaa1e869424e5791881da98e23efb51325ad404aab967809a54239"}
Jul 28 13:35:31.286823 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.286773    2576 scope.go:117] "RemoveContainer" containerID="f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396"
Jul 28 13:35:31.296068 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.296053    2576 scope.go:117] "RemoveContainer" containerID="fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f"
Jul 28 13:35:31.303901 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.303890    2576 scope.go:117] "RemoveContainer" containerID="7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e"
Jul 28 13:35:31.309536 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.309513    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"]
Jul 28 13:35:31.311146 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.311130    2576 scope.go:117] "RemoveContainer" containerID="f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396"
Jul 28 13:35:31.311589 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:35:31.311566    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396\": container with ID starting with f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396 not found: ID does not exist" containerID="f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396"
Jul 28 13:35:31.311676 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.311597    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396"} err="failed to get container status \"f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396\": rpc error: code = NotFound desc = could not find container \"f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396\": container with ID starting with f3c4757dd3648b005f1bede2779f385a5e8bbfaea80f9cbe2ea827d2081b4396 not found: ID does not exist"
Jul 28 13:35:31.311676 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.311616    2576 scope.go:117] "RemoveContainer" containerID="fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f"
Jul 28 13:35:31.311877 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:35:31.311860    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f\": container with ID starting with fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f not found: ID does not exist" containerID="fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f"
Jul 28 13:35:31.311922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.311882    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f"} err="failed to get container status \"fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f\": rpc error: code = NotFound desc = could not find container \"fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f\": container with ID starting with fbde8e23706942c9e544e2e1fa433315800773be2d551464422f16a2bf5af52f not found: ID does not exist"
Jul 28 13:35:31.311922 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.311897    2576 scope.go:117] "RemoveContainer" containerID="7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e"
Jul 28 13:35:31.312105 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:35:31.312089    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e\": container with ID starting with 7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e not found: ID does not exist" containerID="7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e"
Jul 28 13:35:31.312150 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.312108    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e"} err="failed to get container status \"7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e\": rpc error: code = NotFound desc = could not find container \"7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e\": container with ID starting with 7c2c913053bb6034194b5dc16836c398072fc713504bb102700f8748b025164e not found: ID does not exist"
Jul 28 13:35:31.314876 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.314856    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wskww"]
Jul 28 13:35:31.719962 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:35:31.719929    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" path="/var/lib/kubelet/pods/ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde/volumes"
Jul 28 13:43:25.907832 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.907801    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"]
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908159    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container"
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908175    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container"
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908187    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="storage-initializer"
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908193    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="storage-initializer"
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908208    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kube-rbac-proxy"
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908214    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kube-rbac-proxy"
Jul 28 13:43:25.908292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908288    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kube-rbac-proxy"
Jul 28 13:43:25.908588 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.908300    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5ba7aa-5d4e-4e5f-9a67-8369d31e1fde" containerName="kserve-container"
Jul 28 13:43:25.911527 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.911513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:25.915321 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.915301    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:43:25.916606 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.916586    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 13:43:25.916690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.916615    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:43:25.916690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.916615    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:43:25.916690 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.916620    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:43:25.916942 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.916924    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 13:43:25.917166 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.917149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:43:25.929521 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:25.929498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"]
Jul 28 13:43:26.051769 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.051738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t476v\" (UniqueName: \"kubernetes.io/projected/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-api-access-t476v\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.051893 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.051773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.051893 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.051798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125dc9e5-7305-49aa-a985-e0de59a01cb6-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.051893 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.051884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125dc9e5-7305-49aa-a985-e0de59a01cb6-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.052004 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.051916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.152704 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.152669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.152828 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.152744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t476v\" (UniqueName: \"kubernetes.io/projected/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-api-access-t476v\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.152828 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.152761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.152828 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.152778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125dc9e5-7305-49aa-a985-e0de59a01cb6-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.152828 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.152805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125dc9e5-7305-49aa-a985-e0de59a01cb6-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.153223 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.153202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125dc9e5-7305-49aa-a985-e0de59a01cb6-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.153508 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.153485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.153597 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.153485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.155502 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.155479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125dc9e5-7305-49aa-a985-e0de59a01cb6-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.162008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.161960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t476v\" (UniqueName: \"kubernetes.io/projected/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-api-access-t476v\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.222660 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.222629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:26.349865 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:26.349821    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"]
Jul 28 13:43:26.352091 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:43:26.352064    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125dc9e5_7305_49aa_a985_e0de59a01cb6.slice/crio-0c0368adcf5c7bc2b518f8632603e24b6f9a2dac9bfdc31f5c66976c0def331b WatchSource:0}: Error finding container 0c0368adcf5c7bc2b518f8632603e24b6f9a2dac9bfdc31f5c66976c0def331b: Status 404 returned error can't find the container with id 0c0368adcf5c7bc2b518f8632603e24b6f9a2dac9bfdc31f5c66976c0def331b
Jul 28 13:43:27.283561 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:27.283526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerStarted","Data":"acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5"}
Jul 28 13:43:27.284015 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:27.283591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerStarted","Data":"0c0368adcf5c7bc2b518f8632603e24b6f9a2dac9bfdc31f5c66976c0def331b"}
Jul 28 13:43:30.298199 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:30.298118    2576 generic.go:356] "Generic (PLEG): container finished" podID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerID="acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5" exitCode=0
Jul 28 13:43:30.298605 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:30.298192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerDied","Data":"acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5"}
Jul 28 13:43:31.304831 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:31.304798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerStarted","Data":"b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116"}
Jul 28 13:43:31.305214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:31.304862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerStarted","Data":"b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e"}
Jul 28 13:43:31.305214 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:31.305070    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:31.327164 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:31.327120    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" podStartSLOduration=2.683425603 podStartE2EDuration="6.327108625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.643683022s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:31.32613275 +0000 UTC m=+4104.227277141" watchObservedRunningTime="2026-07-28 13:43:31.327108625 +0000 UTC m=+4104.228253080"
Jul 28 13:43:32.309132 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:32.309101    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:43:38.319239 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:43:38.319209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:44:08.390904 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:08.390863    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 13:44:18.322457 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:18.322429    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:44:25.862195 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:25.862165    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"]
Jul 28 13:44:25.862629 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:25.862400    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kserve-container" containerID="cri-o://b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e" gracePeriod=30
Jul 28 13:44:25.862629 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:25.862451    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kube-rbac-proxy" containerID="cri-o://b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116" gracePeriod=30
Jul 28 13:44:26.541185 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:26.541151    2576 generic.go:356] "Generic (PLEG): container finished" podID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerID="b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116" exitCode=2
Jul 28 13:44:26.541347 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:26.541225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerDied","Data":"b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116"}
Jul 28 13:44:28.314619 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:28.314581    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 28 13:44:31.513736 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.513711    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:44:31.562236 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.562180    2576 generic.go:356] "Generic (PLEG): container finished" podID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerID="b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e" exitCode=0
Jul 28 13:44:31.562341 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.562236    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"
Jul 28 13:44:31.562341 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.562261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerDied","Data":"b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e"}
Jul 28 13:44:31.562341 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.562315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb" event={"ID":"125dc9e5-7305-49aa-a985-e0de59a01cb6","Type":"ContainerDied","Data":"0c0368adcf5c7bc2b518f8632603e24b6f9a2dac9bfdc31f5c66976c0def331b"}
Jul 28 13:44:31.562467 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.562347    2576 scope.go:117] "RemoveContainer" containerID="b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116"
Jul 28 13:44:31.570129 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.570116    2576 scope.go:117] "RemoveContainer" containerID="b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e"
Jul 28 13:44:31.577394 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.577364    2576 scope.go:117] "RemoveContainer" containerID="acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5"
Jul 28 13:44:31.584362 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.584346    2576 scope.go:117] "RemoveContainer" containerID="b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116"
Jul 28 13:44:31.584581 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:44:31.584564    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116\": container with ID starting with b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116 not found: ID does not exist" containerID="b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116"
Jul 28 13:44:31.584678 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.584588    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116"} err="failed to get container status \"b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116\": rpc error: code = NotFound desc = could not find container \"b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116\": container with ID starting with b04fd523e1bd5b462e9b66de7dfe19d3ac913afac8677317f9f4cdfd4dff5116 not found: ID does not exist"
Jul 28 13:44:31.584678 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.584607    2576 scope.go:117] "RemoveContainer" containerID="b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e"
Jul 28 13:44:31.584842 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:44:31.584825    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e\": container with ID starting with b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e not found: ID does not exist" containerID="b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e"
Jul 28 13:44:31.584880 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.584849    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e"} err="failed to get container status \"b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e\": rpc error: code = NotFound desc = could not find container \"b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e\": container with ID starting with b35b8137a8f71c5928cf6497b07133a13990d39ed1814acab8c551aa9787fd4e not found: ID does not exist"
Jul 28 13:44:31.584880 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.584864    2576 scope.go:117] "RemoveContainer" containerID="acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5"
Jul 28 13:44:31.585055 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:44:31.585041    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5\": container with ID starting with acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5 not found: ID does not exist" containerID="acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5"
Jul 28 13:44:31.585092 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.585058    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5"} err="failed to get container status \"acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5\": rpc error: code = NotFound desc = could not find container \"acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5\": container with ID starting with acc6ce9578fc89651bfa24edbd01d77752cd9624f56db220598ff37a69d4beb5 not found: ID does not exist"
Jul 28 13:44:31.648588 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.648568    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125dc9e5-7305-49aa-a985-e0de59a01cb6-proxy-tls\") pod \"125dc9e5-7305-49aa-a985-e0de59a01cb6\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") "
Jul 28 13:44:31.648669 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.648610    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t476v\" (UniqueName: \"kubernetes.io/projected/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-api-access-t476v\") pod \"125dc9e5-7305-49aa-a985-e0de59a01cb6\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") "
Jul 28 13:44:31.648710 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.648689    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125dc9e5-7305-49aa-a985-e0de59a01cb6-kserve-provision-location\") pod \"125dc9e5-7305-49aa-a985-e0de59a01cb6\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") "
Jul 28 13:44:31.648755 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.648727    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-rbac-proxy-sar-config\") pod \"125dc9e5-7305-49aa-a985-e0de59a01cb6\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") "
Jul 28 13:44:31.648799 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.648765    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-cabundle-cert\") pod \"125dc9e5-7305-49aa-a985-e0de59a01cb6\" (UID: \"125dc9e5-7305-49aa-a985-e0de59a01cb6\") "
Jul 28 13:44:31.649090 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.649062    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125dc9e5-7305-49aa-a985-e0de59a01cb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "125dc9e5-7305-49aa-a985-e0de59a01cb6" (UID: "125dc9e5-7305-49aa-a985-e0de59a01cb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:44:31.649189 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.649090    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "125dc9e5-7305-49aa-a985-e0de59a01cb6" (UID: "125dc9e5-7305-49aa-a985-e0de59a01cb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:44:31.649189 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.649140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "125dc9e5-7305-49aa-a985-e0de59a01cb6" (UID: "125dc9e5-7305-49aa-a985-e0de59a01cb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:44:31.650619 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.650600    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-api-access-t476v" (OuterVolumeSpecName: "kube-api-access-t476v") pod "125dc9e5-7305-49aa-a985-e0de59a01cb6" (UID: "125dc9e5-7305-49aa-a985-e0de59a01cb6"). InnerVolumeSpecName "kube-api-access-t476v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:44:31.650686 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.650661    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125dc9e5-7305-49aa-a985-e0de59a01cb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "125dc9e5-7305-49aa-a985-e0de59a01cb6" (UID: "125dc9e5-7305-49aa-a985-e0de59a01cb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:44:31.749664 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.749642    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/125dc9e5-7305-49aa-a985-e0de59a01cb6-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:44:31.749664 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.749661    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:44:31.749792 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.749670    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/125dc9e5-7305-49aa-a985-e0de59a01cb6-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:44:31.749792 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.749680    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/125dc9e5-7305-49aa-a985-e0de59a01cb6-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:44:31.749792 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.749688    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t476v\" (UniqueName: \"kubernetes.io/projected/125dc9e5-7305-49aa-a985-e0de59a01cb6-kube-api-access-t476v\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:44:31.879934 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.879909    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"]
Jul 28 13:44:31.883525 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:31.883504    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-w6nkb"]
Jul 28 13:44:33.720767 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:44:33.720730    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" path="/var/lib/kubelet/pods/125dc9e5-7305-49aa-a985-e0de59a01cb6/volumes"
Jul 28 13:45:08.123852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.123822    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"]
Jul 28 13:45:08.124197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124141    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="storage-initializer"
Jul 28 13:45:08.124197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124151    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="storage-initializer"
Jul 28 13:45:08.124197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124169    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kserve-container"
Jul 28 13:45:08.124197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124174    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kserve-container"
Jul 28 13:45:08.124197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124185    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kube-rbac-proxy"
Jul 28 13:45:08.124197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124191    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kube-rbac-proxy"
Jul 28 13:45:08.124422 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124239    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kserve-container"
Jul 28 13:45:08.124422 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.124248    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="125dc9e5-7305-49aa-a985-e0de59a01cb6" containerName="kube-rbac-proxy"
Jul 28 13:45:08.128350 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.128331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.131072 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.131050    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:45:08.131233 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.131208    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:45:08.131363 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.131249    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:45:08.131363 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.131143    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:45:08.131363 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.131174    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 13:45:08.131735 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.131713    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:45:08.132188 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.132170    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:45:08.138226 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.138207    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"]
Jul 28 13:45:08.219165 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.219141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644d5fd1-e984-419b-922e-346dbc84dc22-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.219275 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.219175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.219275 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.219198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.219363 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.219312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6g9s\" (UniqueName: \"kubernetes.io/projected/644d5fd1-e984-419b-922e-346dbc84dc22-kube-api-access-x6g9s\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.219363 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.219336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644d5fd1-e984-419b-922e-346dbc84dc22-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.319955 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.319931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6g9s\" (UniqueName: \"kubernetes.io/projected/644d5fd1-e984-419b-922e-346dbc84dc22-kube-api-access-x6g9s\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320063 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.319962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644d5fd1-e984-419b-922e-346dbc84dc22-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320063 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.319996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644d5fd1-e984-419b-922e-346dbc84dc22-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320063 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.320022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320063 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.320045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320388 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.320356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644d5fd1-e984-419b-922e-346dbc84dc22-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320695 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.320674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.320763 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.320693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.322498 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.322479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644d5fd1-e984-419b-922e-346dbc84dc22-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.328227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.328207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6g9s\" (UniqueName: \"kubernetes.io/projected/644d5fd1-e984-419b-922e-346dbc84dc22-kube-api-access-x6g9s\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.440573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.440521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:08.568805 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.568780    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"]
Jul 28 13:45:08.570846 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:45:08.570821    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644d5fd1_e984_419b_922e_346dbc84dc22.slice/crio-a93cfa24ab8de60ad3ec7a34a1279ee885f0e905c8f0842aebde712dae6d4f7e WatchSource:0}: Error finding container a93cfa24ab8de60ad3ec7a34a1279ee885f0e905c8f0842aebde712dae6d4f7e: Status 404 returned error can't find the container with id a93cfa24ab8de60ad3ec7a34a1279ee885f0e905c8f0842aebde712dae6d4f7e
Jul 28 13:45:08.724149 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.724057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerStarted","Data":"6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef"}
Jul 28 13:45:08.724149 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:08.724127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerStarted","Data":"a93cfa24ab8de60ad3ec7a34a1279ee885f0e905c8f0842aebde712dae6d4f7e"}
Jul 28 13:45:12.744509 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:12.744471    2576 generic.go:356] "Generic (PLEG): container finished" podID="644d5fd1-e984-419b-922e-346dbc84dc22" containerID="6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef" exitCode=0
Jul 28 13:45:12.744984 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:12.744504    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerDied","Data":"6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef"}
Jul 28 13:45:13.750355 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:13.750326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerStarted","Data":"5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735"}
Jul 28 13:45:13.750852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:13.750400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerStarted","Data":"8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f"}
Jul 28 13:45:13.750852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:13.750643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:13.750852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:13.750702    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:13.773604 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:13.773560    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podStartSLOduration=2.122663013 podStartE2EDuration="5.773547032s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650884019s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:45:13.770757764 +0000 UTC m=+4206.671902157" watchObservedRunningTime="2026-07-28 13:45:13.773547032 +0000 UTC m=+4206.674691421"
Jul 28 13:45:19.759852 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:19.759824    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:45:49.790099 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:49.790054    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 13:45:59.762672 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:45:59.762641    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:46:08.251362 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:08.251323    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"]
Jul 28 13:46:08.251864 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:08.251585    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kserve-container" containerID="cri-o://8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f" gracePeriod=30
Jul 28 13:46:08.251864 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:08.251644    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kube-rbac-proxy" containerID="cri-o://5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735" gracePeriod=30
Jul 28 13:46:08.992624 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:08.992588    2576 generic.go:356] "Generic (PLEG): container finished" podID="644d5fd1-e984-419b-922e-346dbc84dc22" containerID="5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735" exitCode=2
Jul 28 13:46:08.992784 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:08.992669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerDied","Data":"5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735"}
Jul 28 13:46:09.754985 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:09.754944    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 28 13:46:10.800563 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:10.800522    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 13:46:14.755609 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:14.755574    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 28 13:46:15.696567 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.696541    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:46:15.832379 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832352    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-cabundle-cert\") pod \"644d5fd1-e984-419b-922e-346dbc84dc22\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") "
Jul 28 13:46:15.832766 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644d5fd1-e984-419b-922e-346dbc84dc22-kserve-provision-location\") pod \"644d5fd1-e984-419b-922e-346dbc84dc22\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") "
Jul 28 13:46:15.832766 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644d5fd1-e984-419b-922e-346dbc84dc22-proxy-tls\") pod \"644d5fd1-e984-419b-922e-346dbc84dc22\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") "
Jul 28 13:46:15.832766 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832457    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6g9s\" (UniqueName: \"kubernetes.io/projected/644d5fd1-e984-419b-922e-346dbc84dc22-kube-api-access-x6g9s\") pod \"644d5fd1-e984-419b-922e-346dbc84dc22\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") "
Jul 28 13:46:15.832766 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832482    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-kube-rbac-proxy-sar-config\") pod \"644d5fd1-e984-419b-922e-346dbc84dc22\" (UID: \"644d5fd1-e984-419b-922e-346dbc84dc22\") "
Jul 28 13:46:15.832923 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832768    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "644d5fd1-e984-419b-922e-346dbc84dc22" (UID: "644d5fd1-e984-419b-922e-346dbc84dc22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:46:15.832923 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832791    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644d5fd1-e984-419b-922e-346dbc84dc22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "644d5fd1-e984-419b-922e-346dbc84dc22" (UID: "644d5fd1-e984-419b-922e-346dbc84dc22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:46:15.832923 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.832858    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "644d5fd1-e984-419b-922e-346dbc84dc22" (UID: "644d5fd1-e984-419b-922e-346dbc84dc22"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:46:15.834619 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.834594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644d5fd1-e984-419b-922e-346dbc84dc22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "644d5fd1-e984-419b-922e-346dbc84dc22" (UID: "644d5fd1-e984-419b-922e-346dbc84dc22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:46:15.834619 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.834606    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644d5fd1-e984-419b-922e-346dbc84dc22-kube-api-access-x6g9s" (OuterVolumeSpecName: "kube-api-access-x6g9s") pod "644d5fd1-e984-419b-922e-346dbc84dc22" (UID: "644d5fd1-e984-419b-922e-346dbc84dc22"). InnerVolumeSpecName "kube-api-access-x6g9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:46:15.933950 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.933924    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:46:15.933950 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.933946    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/644d5fd1-e984-419b-922e-346dbc84dc22-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:46:15.934082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.933960    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/644d5fd1-e984-419b-922e-346dbc84dc22-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:46:15.934082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.933973    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6g9s\" (UniqueName: \"kubernetes.io/projected/644d5fd1-e984-419b-922e-346dbc84dc22-kube-api-access-x6g9s\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:46:15.934082 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:15.933984    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/644d5fd1-e984-419b-922e-346dbc84dc22-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:46:16.022785 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.022755    2576 generic.go:356] "Generic (PLEG): container finished" podID="644d5fd1-e984-419b-922e-346dbc84dc22" containerID="8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f" exitCode=0
Jul 28 13:46:16.022880 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.022811    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"
Jul 28 13:46:16.022880 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.022834    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerDied","Data":"8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f"}
Jul 28 13:46:16.022973 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.022883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx" event={"ID":"644d5fd1-e984-419b-922e-346dbc84dc22","Type":"ContainerDied","Data":"a93cfa24ab8de60ad3ec7a34a1279ee885f0e905c8f0842aebde712dae6d4f7e"}
Jul 28 13:46:16.022973 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.022910    2576 scope.go:117] "RemoveContainer" containerID="5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735"
Jul 28 13:46:16.031941 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.031924    2576 scope.go:117] "RemoveContainer" containerID="8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f"
Jul 28 13:46:16.039424 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.039405    2576 scope.go:117] "RemoveContainer" containerID="6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef"
Jul 28 13:46:16.046604 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.046580    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"]
Jul 28 13:46:16.047152 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.047137    2576 scope.go:117] "RemoveContainer" containerID="5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735"
Jul 28 13:46:16.047404 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:46:16.047362    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735\": container with ID starting with 5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735 not found: ID does not exist" containerID="5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735"
Jul 28 13:46:16.047469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.047403    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735"} err="failed to get container status \"5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735\": rpc error: code = NotFound desc = could not find container \"5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735\": container with ID starting with 5f1fbfbc2a239e43b6964238fda056789e02e0e3ce2e9e229d04b5573199b735 not found: ID does not exist"
Jul 28 13:46:16.047469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.047420    2576 scope.go:117] "RemoveContainer" containerID="8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f"
Jul 28 13:46:16.047715 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:46:16.047684    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f\": container with ID starting with 8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f not found: ID does not exist" containerID="8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f"
Jul 28 13:46:16.047851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.047719    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f"} err="failed to get container status \"8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f\": rpc error: code = NotFound desc = could not find container \"8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f\": container with ID starting with 8280a00e9f8360da1b4cb315ecea3f53a067be8f4982fa95ad14d9cc1048547f not found: ID does not exist"
Jul 28 13:46:16.047851 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.047745    2576 scope.go:117] "RemoveContainer" containerID="6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef"
Jul 28 13:46:16.048116 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:46:16.048077    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef\": container with ID starting with 6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef not found: ID does not exist" containerID="6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef"
Jul 28 13:46:16.048116 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.048108    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef"} err="failed to get container status \"6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef\": rpc error: code = NotFound desc = could not find container \"6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef\": container with ID starting with 6895f850049f6e319bb94dcce47866045061cb9889c26e4a3a6a463b80f90aef not found: ID does not exist"
Jul 28 13:46:16.050223 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:16.050200    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-782vx"]
Jul 28 13:46:17.719969 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:46:17.719941    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" path="/var/lib/kubelet/pods/644d5fd1-e984-419b-922e-346dbc84dc22/volumes"
Jul 28 13:54:48.516421 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516385    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"]
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516850    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kserve-container"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516866    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kserve-container"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516874    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kube-rbac-proxy"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516879    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kube-rbac-proxy"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516900    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="storage-initializer"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516906    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="storage-initializer"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516962    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kserve-container"
Jul 28 13:54:48.518477 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.516973    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="644d5fd1-e984-419b-922e-346dbc84dc22" containerName="kube-rbac-proxy"
Jul 28 13:54:48.520117 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.520099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.523180 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.523162    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 13:54:48.523292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.523201    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:54:48.523292 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.523230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:54:48.524522 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.524508    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:54:48.524624 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.524557    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 13:54:48.524624 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.524578    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:54:48.524813 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.524798    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:54:48.532543 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.532520    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"]
Jul 28 13:54:48.570337 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.570310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq744\" (UniqueName: \"kubernetes.io/projected/2829b305-02eb-412c-aef6-3e7de593a648-kube-api-access-dq744\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.570456 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.570343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.570456 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.570389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2829b305-02eb-412c-aef6-3e7de593a648-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.570456 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.570441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.570591 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.570486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2829b305-02eb-412c-aef6-3e7de593a648-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.670978 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.670951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671069 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.670994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2829b305-02eb-412c-aef6-3e7de593a648-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671069 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.671038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq744\" (UniqueName: \"kubernetes.io/projected/2829b305-02eb-412c-aef6-3e7de593a648-kube-api-access-dq744\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671069 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.671055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671225 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.671081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2829b305-02eb-412c-aef6-3e7de593a648-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671453 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.671438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2829b305-02eb-412c-aef6-3e7de593a648-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.671623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.671691 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.671659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.673540 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.673519    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2829b305-02eb-412c-aef6-3e7de593a648-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.679482 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.679465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq744\" (UniqueName: \"kubernetes.io/projected/2829b305-02eb-412c-aef6-3e7de593a648-kube-api-access-dq744\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.830532 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.830513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:48.961394 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:48.961356    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"]
Jul 28 13:54:48.963329 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:54:48.963304    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2829b305_02eb_412c_aef6_3e7de593a648.slice/crio-d3e36bce41b46bde99292860cd300b4f04d71cb22d3a9d136b794ae08ed35a78 WatchSource:0}: Error finding container d3e36bce41b46bde99292860cd300b4f04d71cb22d3a9d136b794ae08ed35a78: Status 404 returned error can't find the container with id d3e36bce41b46bde99292860cd300b4f04d71cb22d3a9d136b794ae08ed35a78
Jul 28 13:54:49.231929 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:49.231852    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerStarted","Data":"12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa"}
Jul 28 13:54:49.231929 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:49.231917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerStarted","Data":"d3e36bce41b46bde99292860cd300b4f04d71cb22d3a9d136b794ae08ed35a78"}
Jul 28 13:54:53.254639 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:53.254599    2576 generic.go:356] "Generic (PLEG): container finished" podID="2829b305-02eb-412c-aef6-3e7de593a648" containerID="12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa" exitCode=0
Jul 28 13:54:53.255019 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:53.254678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerDied","Data":"12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa"}
Jul 28 13:54:54.261267 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:54.261232    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerStarted","Data":"a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659"}
Jul 28 13:54:54.261667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:54.261290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerStarted","Data":"8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292"}
Jul 28 13:54:54.261667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:54.261479    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:54.261667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:54.261521    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:54:54.283224 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:54:54.283184    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" podStartSLOduration=2.730607966 podStartE2EDuration="6.283170906s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.55256294s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:54:54.281363505 +0000 UTC m=+4787.182507896" watchObservedRunningTime="2026-07-28 13:54:54.283170906 +0000 UTC m=+4787.184315294"
Jul 28 13:55:00.271074 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:00.271047    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:55:30.275955 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:30.275920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:55:38.639069 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:38.639031    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"]
Jul 28 13:55:38.639620 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:38.639458    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kserve-container" containerID="cri-o://8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292" gracePeriod=30
Jul 28 13:55:38.639849 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:38.639801    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kube-rbac-proxy" containerID="cri-o://a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659" gracePeriod=30
Jul 28 13:55:39.452845 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:39.452812    2576 generic.go:356] "Generic (PLEG): container finished" podID="2829b305-02eb-412c-aef6-3e7de593a648" containerID="a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659" exitCode=2
Jul 28 13:55:39.452845 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:39.452851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerDied","Data":"a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659"}
Jul 28 13:55:40.266632 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:40.266588    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 28 13:55:43.689059 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.689036    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:55:43.785858 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.785785    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2829b305-02eb-412c-aef6-3e7de593a648-proxy-tls\") pod \"2829b305-02eb-412c-aef6-3e7de593a648\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") "
Jul 28 13:55:43.785858 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.785835    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2829b305-02eb-412c-aef6-3e7de593a648-kserve-provision-location\") pod \"2829b305-02eb-412c-aef6-3e7de593a648\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") "
Jul 28 13:55:43.786155 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.785889    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-kube-rbac-proxy-sar-config\") pod \"2829b305-02eb-412c-aef6-3e7de593a648\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") "
Jul 28 13:55:43.786155 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.785925    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq744\" (UniqueName: \"kubernetes.io/projected/2829b305-02eb-412c-aef6-3e7de593a648-kube-api-access-dq744\") pod \"2829b305-02eb-412c-aef6-3e7de593a648\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") "
Jul 28 13:55:43.786155 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.785950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-cabundle-cert\") pod \"2829b305-02eb-412c-aef6-3e7de593a648\" (UID: \"2829b305-02eb-412c-aef6-3e7de593a648\") "
Jul 28 13:55:43.786323 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.786226    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2829b305-02eb-412c-aef6-3e7de593a648-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2829b305-02eb-412c-aef6-3e7de593a648" (UID: "2829b305-02eb-412c-aef6-3e7de593a648"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:55:43.786323 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.786250    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2829b305-02eb-412c-aef6-3e7de593a648" (UID: "2829b305-02eb-412c-aef6-3e7de593a648"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:55:43.786469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.786356    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2829b305-02eb-412c-aef6-3e7de593a648" (UID: "2829b305-02eb-412c-aef6-3e7de593a648"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:55:43.788081 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.788048    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2829b305-02eb-412c-aef6-3e7de593a648-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2829b305-02eb-412c-aef6-3e7de593a648" (UID: "2829b305-02eb-412c-aef6-3e7de593a648"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:55:43.788206 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.788146    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2829b305-02eb-412c-aef6-3e7de593a648-kube-api-access-dq744" (OuterVolumeSpecName: "kube-api-access-dq744") pod "2829b305-02eb-412c-aef6-3e7de593a648" (UID: "2829b305-02eb-412c-aef6-3e7de593a648"). InnerVolumeSpecName "kube-api-access-dq744". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:55:43.817775 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.817743    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"]
Jul 28 13:55:43.818151 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818132    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kserve-container"
Jul 28 13:55:43.818151 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818151    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kserve-container"
Jul 28 13:55:43.818284 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818161    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kube-rbac-proxy"
Jul 28 13:55:43.818284 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818167    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kube-rbac-proxy"
Jul 28 13:55:43.818284 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818179    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="storage-initializer"
Jul 28 13:55:43.818284 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818184    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="storage-initializer"
Jul 28 13:55:43.818284 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818260    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kserve-container"
Jul 28 13:55:43.818284 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.818272    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2829b305-02eb-412c-aef6-3e7de593a648" containerName="kube-rbac-proxy"
Jul 28 13:55:43.821558 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.821541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.824249 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.824225    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 13:55:43.824343 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.824290    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 13:55:43.835701 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.835676    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"]
Jul 28 13:55:43.887295 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.887469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.887469 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftgd\" (UniqueName: \"kubernetes.io/projected/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-api-access-tftgd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.887573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.887573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.887654 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887608    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2829b305-02eb-412c-aef6-3e7de593a648-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:55:43.887654 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887633    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2829b305-02eb-412c-aef6-3e7de593a648-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:55:43.887654 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887645    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:55:43.887794 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887657    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq744\" (UniqueName: \"kubernetes.io/projected/2829b305-02eb-412c-aef6-3e7de593a648-kube-api-access-dq744\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:55:43.887794 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.887669    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2829b305-02eb-412c-aef6-3e7de593a648-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:55:43.988930 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.988902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftgd\" (UniqueName: \"kubernetes.io/projected/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-api-access-tftgd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989092 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.988935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989092 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.988956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989092 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.988988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989092 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.989014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989366 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.989346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989656 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.989639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.989709 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.989693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.991440 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.991423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:43.998047 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:43.998022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftgd\" (UniqueName: \"kubernetes.io/projected/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-api-access-tftgd\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-94l97\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:44.134850 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.134828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:44.266915 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.266886    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"]
Jul 28 13:55:44.268826 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:55:44.268803    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc501b51_1f12_4b12_807b_0c4cd8fe3cff.slice/crio-e5328f56ce2a82b74cef6ca61dca96c993588d2061d8cfb9b05ab9d629c447cc WatchSource:0}: Error finding container e5328f56ce2a82b74cef6ca61dca96c993588d2061d8cfb9b05ab9d629c447cc: Status 404 returned error can't find the container with id e5328f56ce2a82b74cef6ca61dca96c993588d2061d8cfb9b05ab9d629c447cc
Jul 28 13:55:44.476489 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.476417    2576 generic.go:356] "Generic (PLEG): container finished" podID="2829b305-02eb-412c-aef6-3e7de593a648" containerID="8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292" exitCode=0
Jul 28 13:55:44.476635 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.476489    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"
Jul 28 13:55:44.476635 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.476506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerDied","Data":"8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292"}
Jul 28 13:55:44.476635 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.476575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g" event={"ID":"2829b305-02eb-412c-aef6-3e7de593a648","Type":"ContainerDied","Data":"d3e36bce41b46bde99292860cd300b4f04d71cb22d3a9d136b794ae08ed35a78"}
Jul 28 13:55:44.476635 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.476615    2576 scope.go:117] "RemoveContainer" containerID="a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659"
Jul 28 13:55:44.478299 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.478274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerStarted","Data":"9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf"}
Jul 28 13:55:44.478429 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.478335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerStarted","Data":"e5328f56ce2a82b74cef6ca61dca96c993588d2061d8cfb9b05ab9d629c447cc"}
Jul 28 13:55:44.485227 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.485210    2576 scope.go:117] "RemoveContainer" containerID="8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292"
Jul 28 13:55:44.492412 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.492393    2576 scope.go:117] "RemoveContainer" containerID="12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa"
Jul 28 13:55:44.499190 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.499180    2576 scope.go:117] "RemoveContainer" containerID="a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659"
Jul 28 13:55:44.499506 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:55:44.499488    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659\": container with ID starting with a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659 not found: ID does not exist" containerID="a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659"
Jul 28 13:55:44.499604 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.499518    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659"} err="failed to get container status \"a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659\": rpc error: code = NotFound desc = could not find container \"a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659\": container with ID starting with a13675ba2e9eb05140a4164a4d396e394fabb29b377c10462394205c8cae5659 not found: ID does not exist"
Jul 28 13:55:44.499604 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.499535    2576 scope.go:117] "RemoveContainer" containerID="8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292"
Jul 28 13:55:44.499785 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:55:44.499765    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292\": container with ID starting with 8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292 not found: ID does not exist" containerID="8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292"
Jul 28 13:55:44.499857 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.499791    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292"} err="failed to get container status \"8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292\": rpc error: code = NotFound desc = could not find container \"8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292\": container with ID starting with 8fd9af543de41f4e3dc2bcd57c491525d83eeb5471371d52f712e0d103ae1292 not found: ID does not exist"
Jul 28 13:55:44.499857 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.499806    2576 scope.go:117] "RemoveContainer" containerID="12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa"
Jul 28 13:55:44.500047 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:55:44.500028    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa\": container with ID starting with 12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa not found: ID does not exist" containerID="12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa"
Jul 28 13:55:44.500119 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.500051    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa"} err="failed to get container status \"12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa\": rpc error: code = NotFound desc = could not find container \"12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa\": container with ID starting with 12e34028c12b0d9fb3e251043403f7e7d5cc55273ed4a427ee883db8cda56dfa not found: ID does not exist"
Jul 28 13:55:44.518443 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.518418    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"]
Jul 28 13:55:44.524589 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:44.524565    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fwt2g"]
Jul 28 13:55:45.720684 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:45.720636    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2829b305-02eb-412c-aef6-3e7de593a648" path="/var/lib/kubelet/pods/2829b305-02eb-412c-aef6-3e7de593a648/volumes"
Jul 28 13:55:48.498975 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:48.498890    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerID="9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf" exitCode=0
Jul 28 13:55:48.498975 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:48.498923    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerDied","Data":"9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf"}
Jul 28 13:55:49.506721 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:49.506683    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerStarted","Data":"a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca"}
Jul 28 13:55:49.507138 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:49.506771    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerStarted","Data":"bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc"}
Jul 28 13:55:49.507138 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:49.507006    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:49.529395 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:49.529318    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" podStartSLOduration=2.755766984 podStartE2EDuration="6.529304278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773537294s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:55:49.527183198 +0000 UTC m=+4842.428327589" watchObservedRunningTime="2026-07-28 13:55:49.529304278 +0000 UTC m=+4842.430448668"
Jul 28 13:55:50.512144 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:50.512113    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:55:56.522097 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:55:56.522066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:56:26.526766 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:26.526731    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:56:33.862666 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:33.862634    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"]
Jul 28 13:56:33.863046 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:33.862857    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kserve-container" containerID="cri-o://bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc" gracePeriod=30
Jul 28 13:56:33.863046 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:33.862887    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kube-rbac-proxy" containerID="cri-o://a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca" gracePeriod=30
Jul 28 13:56:34.717036 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:34.717002    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerID="a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca" exitCode=2
Jul 28 13:56:34.717197 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:34.717046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerDied","Data":"a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca"}
Jul 28 13:56:36.517140 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:36.517099    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 13:56:36.522953 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:36.522929    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.56:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 28 13:56:38.704965 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.704942    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:56:38.722830 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.722809    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-cabundle-cert\") pod \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") "
Jul 28 13:56:38.722933 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.722850    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-rbac-proxy-sar-config\") pod \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") "
Jul 28 13:56:38.722933 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.722876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kserve-provision-location\") pod \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") "
Jul 28 13:56:38.723036 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.723014    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tftgd\" (UniqueName: \"kubernetes.io/projected/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-api-access-tftgd\") pod \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") "
Jul 28 13:56:38.723085 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.723064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-proxy-tls\") pod \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\" (UID: \"cc501b51-1f12-4b12-807b-0c4cd8fe3cff\") "
Jul 28 13:56:38.723142 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.723131    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc501b51-1f12-4b12-807b-0c4cd8fe3cff" (UID: "cc501b51-1f12-4b12-807b-0c4cd8fe3cff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:56:38.723250 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.723198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc501b51-1f12-4b12-807b-0c4cd8fe3cff" (UID: "cc501b51-1f12-4b12-807b-0c4cd8fe3cff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:56:38.723250 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.723207    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc501b51-1f12-4b12-807b-0c4cd8fe3cff" (UID: "cc501b51-1f12-4b12-807b-0c4cd8fe3cff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:56:38.724647 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.724613    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:56:38.728695 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.727311    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:56:38.728695 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.727905    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:56:38.728695 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.726405    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc501b51-1f12-4b12-807b-0c4cd8fe3cff" (UID: "cc501b51-1f12-4b12-807b-0c4cd8fe3cff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:56:38.728695 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.727787    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-api-access-tftgd" (OuterVolumeSpecName: "kube-api-access-tftgd") pod "cc501b51-1f12-4b12-807b-0c4cd8fe3cff" (UID: "cc501b51-1f12-4b12-807b-0c4cd8fe3cff"). InnerVolumeSpecName "kube-api-access-tftgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:56:38.742564 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.742514    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerID="bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc" exitCode=0
Jul 28 13:56:38.742653 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.742571    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"
Jul 28 13:56:38.742653 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.742598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerDied","Data":"bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc"}
Jul 28 13:56:38.742737 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.742704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97" event={"ID":"cc501b51-1f12-4b12-807b-0c4cd8fe3cff","Type":"ContainerDied","Data":"e5328f56ce2a82b74cef6ca61dca96c993588d2061d8cfb9b05ab9d629c447cc"}
Jul 28 13:56:38.742774 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.742747    2576 scope.go:117] "RemoveContainer" containerID="a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca"
Jul 28 13:56:38.752036 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.752020    2576 scope.go:117] "RemoveContainer" containerID="bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc"
Jul 28 13:56:38.760554 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.760536    2576 scope.go:117] "RemoveContainer" containerID="9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf"
Jul 28 13:56:38.765693 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.765674    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"]
Jul 28 13:56:38.768357 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.768342    2576 scope.go:117] "RemoveContainer" containerID="a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca"
Jul 28 13:56:38.768615 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:56:38.768597    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca\": container with ID starting with a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca not found: ID does not exist" containerID="a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca"
Jul 28 13:56:38.768674 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.768623    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca"} err="failed to get container status \"a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca\": rpc error: code = NotFound desc = could not find container \"a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca\": container with ID starting with a6cf73f78148d8b9a1b737c92586d2aaaddb3c5248a5095d569f7bbf1a20e2ca not found: ID does not exist"
Jul 28 13:56:38.768674 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.768639    2576 scope.go:117] "RemoveContainer" containerID="bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc"
Jul 28 13:56:38.768882 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:56:38.768867    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc\": container with ID starting with bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc not found: ID does not exist" containerID="bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc"
Jul 28 13:56:38.768932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.768887    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc"} err="failed to get container status \"bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc\": rpc error: code = NotFound desc = could not find container \"bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc\": container with ID starting with bd86f7c0da81968265ad492cf16641ffee418a96c8fb25f118bdc82368716efc not found: ID does not exist"
Jul 28 13:56:38.768932 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.768902    2576 scope.go:117] "RemoveContainer" containerID="9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf"
Jul 28 13:56:38.769145 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:56:38.769126    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf\": container with ID starting with 9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf not found: ID does not exist" containerID="9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf"
Jul 28 13:56:38.769196 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.769150    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf"} err="failed to get container status \"9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf\": rpc error: code = NotFound desc = could not find container \"9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf\": container with ID starting with 9ecd8eb7c25adcad0a5e36a895decdf4b8910b8aaa8a1b4bc1c21c2a4626e9bf not found: ID does not exist"
Jul 28 13:56:38.769902 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.769884    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-94l97"]
Jul 28 13:56:38.829245 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.829217    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftgd\" (UniqueName: \"kubernetes.io/projected/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-kube-api-access-tftgd\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:56:38.829245 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:38.829246    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc501b51-1f12-4b12-807b-0c4cd8fe3cff-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:56:39.720348 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:56:39.720315    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" path="/var/lib/kubelet/pods/cc501b51-1f12-4b12-807b-0c4cd8fe3cff/volumes"
Jul 28 13:58:09.433611 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.433572    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"]
Jul 28 13:58:09.434185 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434126    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kserve-container"
Jul 28 13:58:09.434185 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434168    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kserve-container"
Jul 28 13:58:09.434318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434206    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="storage-initializer"
Jul 28 13:58:09.434318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434215    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="storage-initializer"
Jul 28 13:58:09.434318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434225    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kube-rbac-proxy"
Jul 28 13:58:09.434318 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434235    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kube-rbac-proxy"
Jul 28 13:58:09.434573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434342    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kserve-container"
Jul 28 13:58:09.434573 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.434354    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc501b51-1f12-4b12-807b-0c4cd8fe3cff" containerName="kube-rbac-proxy"
Jul 28 13:58:09.437581 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.437561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.440328 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.440283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-x97q8\""
Jul 28 13:58:09.440456 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.440361    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:58:09.440456 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.440428    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:58:09.440582 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.440432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 13:58:09.441383 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.441357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:58:09.441524 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.441506    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:58:09.441606 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.441539    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:58:09.446483 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.446452    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"]
Jul 28 13:58:09.576221 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.576183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.576341 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.576241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12341c43-4ba8-4806-aefd-811b638d1e45-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.576341 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.576320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12341c43-4ba8-4806-aefd-811b638d1e45-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.576457 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.576349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwv8\" (UniqueName: \"kubernetes.io/projected/12341c43-4ba8-4806-aefd-811b638d1e45-kube-api-access-ptwv8\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.576457 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.576397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.677215 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.677190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12341c43-4ba8-4806-aefd-811b638d1e45-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.677314 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.677245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12341c43-4ba8-4806-aefd-811b638d1e45-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.677314 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.677273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwv8\" (UniqueName: \"kubernetes.io/projected/12341c43-4ba8-4806-aefd-811b638d1e45-kube-api-access-ptwv8\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.677314 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.677294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.677493 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.677319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.677801 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.677774    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12341c43-4ba8-4806-aefd-811b638d1e45-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.678198 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.678174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.678281 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.678174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.679819 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.679793    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12341c43-4ba8-4806-aefd-811b638d1e45-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.685985 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.685919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwv8\" (UniqueName: \"kubernetes.io/projected/12341c43-4ba8-4806-aefd-811b638d1e45-kube-api-access-ptwv8\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.749742 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.749719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:09.882339 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:09.882310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"]
Jul 28 13:58:09.883734 ip-10-0-129-190 kubenswrapper[2576]: W0728 13:58:09.883710    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12341c43_4ba8_4806_aefd_811b638d1e45.slice/crio-8dee4c9cd361b394f8096dc9b3722611062caa9427deda30fb591edb52eb39ae WatchSource:0}: Error finding container 8dee4c9cd361b394f8096dc9b3722611062caa9427deda30fb591edb52eb39ae: Status 404 returned error can't find the container with id 8dee4c9cd361b394f8096dc9b3722611062caa9427deda30fb591edb52eb39ae
Jul 28 13:58:10.145236 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:10.145201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerStarted","Data":"638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c"}
Jul 28 13:58:10.145447 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:10.145262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerStarted","Data":"8dee4c9cd361b394f8096dc9b3722611062caa9427deda30fb591edb52eb39ae"}
Jul 28 13:58:14.164814 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:14.164777    2576 generic.go:356] "Generic (PLEG): container finished" podID="12341c43-4ba8-4806-aefd-811b638d1e45" containerID="638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c" exitCode=0
Jul 28 13:58:14.165180 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:14.164825    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerDied","Data":"638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c"}
Jul 28 13:58:15.171210 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:15.171171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerStarted","Data":"f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f"}
Jul 28 13:58:15.171676 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:15.171238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerStarted","Data":"268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482"}
Jul 28 13:58:15.171676 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:15.171431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:15.191533 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:15.191478    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" podStartSLOduration=2.379946393 podStartE2EDuration="6.191459317s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.811512924s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:58:15.191003758 +0000 UTC m=+4988.092148148" watchObservedRunningTime="2026-07-28 13:58:15.191459317 +0000 UTC m=+4988.092603709"
Jul 28 13:58:16.175517 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:16.175484    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:22.184037 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:22.184007    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:58:52.191085 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:58:52.191043    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 13:59:02.186917 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:02.186878    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:59:09.590476 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:09.590444    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"]
Jul 28 13:59:09.591008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:09.590783    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kserve-container" containerID="cri-o://268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482" gracePeriod=30
Jul 28 13:59:09.591008 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:09.590826    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kube-rbac-proxy" containerID="cri-o://f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f" gracePeriod=30
Jul 28 13:59:10.422155 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:10.422122    2576 generic.go:356] "Generic (PLEG): container finished" podID="12341c43-4ba8-4806-aefd-811b638d1e45" containerID="f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f" exitCode=2
Jul 28 13:59:10.422331 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:10.422195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerDied","Data":"f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f"}
Jul 28 13:59:12.179712 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:12.179672    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jul 28 13:59:15.436988 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.436966    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:59:15.444362 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.444340    2576 generic.go:356] "Generic (PLEG): container finished" podID="12341c43-4ba8-4806-aefd-811b638d1e45" containerID="268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482" exitCode=0
Jul 28 13:59:15.444472 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.444392    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"
Jul 28 13:59:15.444472 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.444403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerDied","Data":"268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482"}
Jul 28 13:59:15.444472 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.444468    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz" event={"ID":"12341c43-4ba8-4806-aefd-811b638d1e45","Type":"ContainerDied","Data":"8dee4c9cd361b394f8096dc9b3722611062caa9427deda30fb591edb52eb39ae"}
Jul 28 13:59:15.444578 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.444505    2576 scope.go:117] "RemoveContainer" containerID="f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f"
Jul 28 13:59:15.452133 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.452117    2576 scope.go:117] "RemoveContainer" containerID="268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482"
Jul 28 13:59:15.461113 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.461098    2576 scope.go:117] "RemoveContainer" containerID="638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c"
Jul 28 13:59:15.469545 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.469526    2576 scope.go:117] "RemoveContainer" containerID="f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f"
Jul 28 13:59:15.469803 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:59:15.469784    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f\": container with ID starting with f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f not found: ID does not exist" containerID="f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f"
Jul 28 13:59:15.469865 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.469812    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f"} err="failed to get container status \"f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f\": rpc error: code = NotFound desc = could not find container \"f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f\": container with ID starting with f2519b7557f892893704000070ec402e9fb971f174f9b409acb8d66997c62a2f not found: ID does not exist"
Jul 28 13:59:15.469865 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.469830    2576 scope.go:117] "RemoveContainer" containerID="268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482"
Jul 28 13:59:15.470049 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:59:15.470033    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482\": container with ID starting with 268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482 not found: ID does not exist" containerID="268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482"
Jul 28 13:59:15.470099 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.470057    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482"} err="failed to get container status \"268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482\": rpc error: code = NotFound desc = could not find container \"268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482\": container with ID starting with 268295eddb9544c2853aca707dd2e260fd5496382257f9acf5dc742cdad94482 not found: ID does not exist"
Jul 28 13:59:15.470099 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.470073    2576 scope.go:117] "RemoveContainer" containerID="638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c"
Jul 28 13:59:15.470319 ip-10-0-129-190 kubenswrapper[2576]: E0728 13:59:15.470301    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c\": container with ID starting with 638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c not found: ID does not exist" containerID="638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c"
Jul 28 13:59:15.470364 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.470323    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c"} err="failed to get container status \"638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c\": rpc error: code = NotFound desc = could not find container \"638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c\": container with ID starting with 638c0d4c507ce325d864abe0a5abaedd5aa39b6c87c2346b2bd0dc0cab436d9c not found: ID does not exist"
Jul 28 13:59:15.573064 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573034    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12341c43-4ba8-4806-aefd-811b638d1e45-kserve-provision-location\") pod \"12341c43-4ba8-4806-aefd-811b638d1e45\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") "
Jul 28 13:59:15.573171 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573078    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptwv8\" (UniqueName: \"kubernetes.io/projected/12341c43-4ba8-4806-aefd-811b638d1e45-kube-api-access-ptwv8\") pod \"12341c43-4ba8-4806-aefd-811b638d1e45\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") "
Jul 28 13:59:15.573171 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-cabundle-cert\") pod \"12341c43-4ba8-4806-aefd-811b638d1e45\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") "
Jul 28 13:59:15.573171 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573137    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12341c43-4ba8-4806-aefd-811b638d1e45-proxy-tls\") pod \"12341c43-4ba8-4806-aefd-811b638d1e45\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") "
Jul 28 13:59:15.573171 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-kube-rbac-proxy-sar-config\") pod \"12341c43-4ba8-4806-aefd-811b638d1e45\" (UID: \"12341c43-4ba8-4806-aefd-811b638d1e45\") "
Jul 28 13:59:15.573409 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573365    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12341c43-4ba8-4806-aefd-811b638d1e45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12341c43-4ba8-4806-aefd-811b638d1e45" (UID: "12341c43-4ba8-4806-aefd-811b638d1e45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:59:15.573513 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573480    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12341c43-4ba8-4806-aefd-811b638d1e45" (UID: "12341c43-4ba8-4806-aefd-811b638d1e45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:59:15.573667 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.573646    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12341c43-4ba8-4806-aefd-811b638d1e45" (UID: "12341c43-4ba8-4806-aefd-811b638d1e45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:59:15.575097 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.575069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12341c43-4ba8-4806-aefd-811b638d1e45-kube-api-access-ptwv8" (OuterVolumeSpecName: "kube-api-access-ptwv8") pod "12341c43-4ba8-4806-aefd-811b638d1e45" (UID: "12341c43-4ba8-4806-aefd-811b638d1e45"). InnerVolumeSpecName "kube-api-access-ptwv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:59:15.575180 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.575134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12341c43-4ba8-4806-aefd-811b638d1e45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12341c43-4ba8-4806-aefd-811b638d1e45" (UID: "12341c43-4ba8-4806-aefd-811b638d1e45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:59:15.674497 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.674475    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12341c43-4ba8-4806-aefd-811b638d1e45-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:59:15.674497 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.674494    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwv8\" (UniqueName: \"kubernetes.io/projected/12341c43-4ba8-4806-aefd-811b638d1e45-kube-api-access-ptwv8\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:59:15.674615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.674503    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:59:15.674615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.674513    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12341c43-4ba8-4806-aefd-811b638d1e45-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:59:15.674615 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.674520    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12341c43-4ba8-4806-aefd-811b638d1e45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 13:59:15.761655 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.761622    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"]
Jul 28 13:59:15.765895 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:15.765874    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v96kz"]
Jul 28 13:59:17.719993 ip-10-0-129-190 kubenswrapper[2576]: I0728 13:59:17.719956    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" path="/var/lib/kubelet/pods/12341c43-4ba8-4806-aefd-811b638d1e45/volumes"
Jul 28 14:06:59.292038 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292006    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b2dp4/must-gather-bsh4s"]
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292318    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="storage-initializer"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292328    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="storage-initializer"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292349    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kube-rbac-proxy"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292354    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kube-rbac-proxy"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292363    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kserve-container"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292382    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kserve-container"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292427    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kserve-container"
Jul 28 14:06:59.292448 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.292438    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12341c43-4ba8-4806-aefd-811b638d1e45" containerName="kube-rbac-proxy"
Jul 28 14:06:59.295602 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.295584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.299296 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.299278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b2dp4\"/\"default-dockercfg-qtxcf\""
Jul 28 14:06:59.299550 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.299536    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2dp4\"/\"openshift-service-ca.crt\""
Jul 28 14:06:59.300928 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.300912    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2dp4\"/\"kube-root-ca.crt\""
Jul 28 14:06:59.306705 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.306686    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2dp4/must-gather-bsh4s"]
Jul 28 14:06:59.451592 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.451569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84912c46-71ad-45b5-b24a-c0df3e872ffd-must-gather-output\") pod \"must-gather-bsh4s\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") " pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.451690 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.451606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62n8k\" (UniqueName: \"kubernetes.io/projected/84912c46-71ad-45b5-b24a-c0df3e872ffd-kube-api-access-62n8k\") pod \"must-gather-bsh4s\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") " pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.552003 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.551956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62n8k\" (UniqueName: \"kubernetes.io/projected/84912c46-71ad-45b5-b24a-c0df3e872ffd-kube-api-access-62n8k\") pod \"must-gather-bsh4s\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") " pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.552089 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.552009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84912c46-71ad-45b5-b24a-c0df3e872ffd-must-gather-output\") pod \"must-gather-bsh4s\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") " pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.552247 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.552232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84912c46-71ad-45b5-b24a-c0df3e872ffd-must-gather-output\") pod \"must-gather-bsh4s\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") " pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.561596 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.561571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62n8k\" (UniqueName: \"kubernetes.io/projected/84912c46-71ad-45b5-b24a-c0df3e872ffd-kube-api-access-62n8k\") pod \"must-gather-bsh4s\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") " pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.610559 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.610535    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:06:59.735357 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.735335    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2dp4/must-gather-bsh4s"]
Jul 28 14:06:59.737956 ip-10-0-129-190 kubenswrapper[2576]: W0728 14:06:59.737929    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84912c46_71ad_45b5_b24a_c0df3e872ffd.slice/crio-1c5787c1d31d9ed41747d79eeeca8987ef6ba3954f7a46f9fac0ebeaf5e8fac4 WatchSource:0}: Error finding container 1c5787c1d31d9ed41747d79eeeca8987ef6ba3954f7a46f9fac0ebeaf5e8fac4: Status 404 returned error can't find the container with id 1c5787c1d31d9ed41747d79eeeca8987ef6ba3954f7a46f9fac0ebeaf5e8fac4
Jul 28 14:06:59.739744 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:06:59.739718    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:07:00.418355 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:00.418323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2dp4/must-gather-bsh4s" event={"ID":"84912c46-71ad-45b5-b24a-c0df3e872ffd","Type":"ContainerStarted","Data":"1c5787c1d31d9ed41747d79eeeca8987ef6ba3954f7a46f9fac0ebeaf5e8fac4"}
Jul 28 14:07:06.450540 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:06.450502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2dp4/must-gather-bsh4s" event={"ID":"84912c46-71ad-45b5-b24a-c0df3e872ffd","Type":"ContainerStarted","Data":"b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead"}
Jul 28 14:07:06.450915 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:06.450552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2dp4/must-gather-bsh4s" event={"ID":"84912c46-71ad-45b5-b24a-c0df3e872ffd","Type":"ContainerStarted","Data":"ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"}
Jul 28 14:07:06.467912 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:06.467869    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b2dp4/must-gather-bsh4s" podStartSLOduration=1.366455399 podStartE2EDuration="7.467854296s" totalImagesPullingTime="6.101398897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:07:06.466171778 +0000 UTC m=+5519.367316169" watchObservedRunningTime="2026-07-28 14:07:06.467854296 +0000 UTC m=+5519.368998684"
Jul 28 14:07:31.570074 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:31.570043    2576 generic.go:356] "Generic (PLEG): container finished" podID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerID="ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792" exitCode=0
Jul 28 14:07:31.570608 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:31.570115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2dp4/must-gather-bsh4s" event={"ID":"84912c46-71ad-45b5-b24a-c0df3e872ffd","Type":"ContainerDied","Data":"ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"}
Jul 28 14:07:31.570608 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:31.570522    2576 scope.go:117] "RemoveContainer" containerID="ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"
Jul 28 14:07:31.741345 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:31.741322    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b2dp4_must-gather-bsh4s_84912c46-71ad-45b5-b24a-c0df3e872ffd/gather/0.log"
Jul 28 14:07:36.510691 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:36.510655    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rg6kq_5352b7da-d4d1-4cca-9d6e-d522d0495f42/global-pull-secret-syncer/0.log"
Jul 28 14:07:36.705356 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:36.705324    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hmz9l_6342dafc-bb6e-4bda-9dcd-9afed70b8a6a/konnectivity-agent/0.log"
Jul 28 14:07:36.783294 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:36.783226    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-190.ec2.internal_1705df2f09f7b34e26c5e6a8a8046a32/haproxy/0.log"
Jul 28 14:07:37.806309 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:37.806274    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-b2dp4/must-gather-bsh4s"]
Jul 28 14:07:37.806722 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:37.806484    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-b2dp4/must-gather-bsh4s" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="copy" containerID="cri-o://b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead" gracePeriod=2
Jul 28 14:07:37.812062 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:37.812035    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-b2dp4/must-gather-bsh4s"]
Jul 28 14:07:38.039167 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.039145    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b2dp4_must-gather-bsh4s_84912c46-71ad-45b5-b24a-c0df3e872ffd/copy/0.log"
Jul 28 14:07:38.041573 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.041557    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:07:38.044087 ip-10-0-129-190 kubenswrapper[2576]: E0728 14:07:38.044068    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bsh4s\" is forbidden: User \"system:node:ip-10-0-129-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2dp4\": no relationship found between node 'ip-10-0-129-190.ec2.internal' and this object" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:07:38.085531 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.085509    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62n8k\" (UniqueName: \"kubernetes.io/projected/84912c46-71ad-45b5-b24a-c0df3e872ffd-kube-api-access-62n8k\") pod \"84912c46-71ad-45b5-b24a-c0df3e872ffd\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") "
Jul 28 14:07:38.085649 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.085619    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84912c46-71ad-45b5-b24a-c0df3e872ffd-must-gather-output\") pod \"84912c46-71ad-45b5-b24a-c0df3e872ffd\" (UID: \"84912c46-71ad-45b5-b24a-c0df3e872ffd\") "
Jul 28 14:07:38.087158 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.087129    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84912c46-71ad-45b5-b24a-c0df3e872ffd-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "84912c46-71ad-45b5-b24a-c0df3e872ffd" (UID: "84912c46-71ad-45b5-b24a-c0df3e872ffd"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:07:38.087673 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.087655    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84912c46-71ad-45b5-b24a-c0df3e872ffd-kube-api-access-62n8k" (OuterVolumeSpecName: "kube-api-access-62n8k") pod "84912c46-71ad-45b5-b24a-c0df3e872ffd" (UID: "84912c46-71ad-45b5-b24a-c0df3e872ffd"). InnerVolumeSpecName "kube-api-access-62n8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:07:38.186955 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.186918    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62n8k\" (UniqueName: \"kubernetes.io/projected/84912c46-71ad-45b5-b24a-c0df3e872ffd-kube-api-access-62n8k\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 14:07:38.186955 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.186948    2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84912c46-71ad-45b5-b24a-c0df3e872ffd-must-gather-output\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Jul 28 14:07:38.603199 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.603174    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-b2dp4_must-gather-bsh4s_84912c46-71ad-45b5-b24a-c0df3e872ffd/copy/0.log"
Jul 28 14:07:38.603537 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.603518    2576 generic.go:356] "Generic (PLEG): container finished" podID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerID="b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead" exitCode=143
Jul 28 14:07:38.603603 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.603570    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:07:38.603650 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.603614    2576 scope.go:117] "RemoveContainer" containerID="b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead"
Jul 28 14:07:38.606142 ip-10-0-129-190 kubenswrapper[2576]: E0728 14:07:38.606112    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bsh4s\" is forbidden: User \"system:node:ip-10-0-129-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2dp4\": no relationship found between node 'ip-10-0-129-190.ec2.internal' and this object" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:07:38.612287 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.612267    2576 scope.go:117] "RemoveContainer" containerID="ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"
Jul 28 14:07:38.614838 ip-10-0-129-190 kubenswrapper[2576]: E0728 14:07:38.614810    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-bsh4s\" is forbidden: User \"system:node:ip-10-0-129-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-b2dp4\": no relationship found between node 'ip-10-0-129-190.ec2.internal' and this object" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" pod="openshift-must-gather-b2dp4/must-gather-bsh4s"
Jul 28 14:07:38.624065 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.624050    2576 scope.go:117] "RemoveContainer" containerID="b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead"
Jul 28 14:07:38.624327 ip-10-0-129-190 kubenswrapper[2576]: E0728 14:07:38.624307    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead\": container with ID starting with b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead not found: ID does not exist" containerID="b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead"
Jul 28 14:07:38.624392 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.624335    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead"} err="failed to get container status \"b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead\": rpc error: code = NotFound desc = could not find container \"b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead\": container with ID starting with b2f01404ef0bbd40f99095769800318f62f81055cb765a80f9128b2b4383cead not found: ID does not exist"
Jul 28 14:07:38.624392 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.624353    2576 scope.go:117] "RemoveContainer" containerID="ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"
Jul 28 14:07:38.624602 ip-10-0-129-190 kubenswrapper[2576]: E0728 14:07:38.624585    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792\": container with ID starting with ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792 not found: ID does not exist" containerID="ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"
Jul 28 14:07:38.624651 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:38.624608    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792"} err="failed to get container status \"ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792\": rpc error: code = NotFound desc = could not find container \"ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792\": container with ID starting with ec0f8fa60f63d1b37fbdc231f4664921397c486a2469c6a69f1f68a42154f792 not found: ID does not exist"
Jul 28 14:07:39.720086 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:39.720053    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" path="/var/lib/kubelet/pods/84912c46-71ad-45b5-b24a-c0df3e872ffd/volumes"
Jul 28 14:07:41.305801 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.305767    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/alertmanager/0.log"
Jul 28 14:07:41.333416 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.333393    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/config-reloader/0.log"
Jul 28 14:07:41.359730 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.359704    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/kube-rbac-proxy-web/0.log"
Jul 28 14:07:41.386122 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.386100    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/kube-rbac-proxy/0.log"
Jul 28 14:07:41.415687 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.415667    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/kube-rbac-proxy-metric/0.log"
Jul 28 14:07:41.444781 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.444762    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/prom-label-proxy/0.log"
Jul 28 14:07:41.469936 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.469918    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e0540da8-e3fe-425f-9634-0791ba686651/init-config-reloader/0.log"
Jul 28 14:07:41.576439 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.576419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-rpv9f_5c2f3458-2fa4-49b5-9c3e-691b1e6276ec/kube-state-metrics/0.log"
Jul 28 14:07:41.607220 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.607197    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-rpv9f_5c2f3458-2fa4-49b5-9c3e-691b1e6276ec/kube-rbac-proxy-main/0.log"
Jul 28 14:07:41.635397 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.635360    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-rpv9f_5c2f3458-2fa4-49b5-9c3e-691b1e6276ec/kube-rbac-proxy-self/0.log"
Jul 28 14:07:41.670158 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.670135    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-56dc799bb-l8mmf_bcce6beb-54b5-46ee-b8de-e6de93704f43/metrics-server/0.log"
Jul 28 14:07:41.703845 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.703823    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-7tmjl_8dc012f0-ae6b-4f69-aedb-59860abacf47/monitoring-plugin/0.log"
Jul 28 14:07:41.841842 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.841783    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l6shh_51b7d56b-f47c-48a5-b2b9-b26451aee178/node-exporter/0.log"
Jul 28 14:07:41.867117 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.867094    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l6shh_51b7d56b-f47c-48a5-b2b9-b26451aee178/kube-rbac-proxy/0.log"
Jul 28 14:07:41.892355 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:41.892329    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l6shh_51b7d56b-f47c-48a5-b2b9-b26451aee178/init-textfile/0.log"
Jul 28 14:07:42.027640 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.027610    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6r5gw_e679bdb4-1b51-4f64-b3b3-2567f68d030d/kube-rbac-proxy-main/0.log"
Jul 28 14:07:42.053922 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.053892    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6r5gw_e679bdb4-1b51-4f64-b3b3-2567f68d030d/kube-rbac-proxy-self/0.log"
Jul 28 14:07:42.082502 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.082464    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6r5gw_e679bdb4-1b51-4f64-b3b3-2567f68d030d/openshift-state-metrics/0.log"
Jul 28 14:07:42.129536 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.129453    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/prometheus/0.log"
Jul 28 14:07:42.154770 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.154747    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/config-reloader/0.log"
Jul 28 14:07:42.192578 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.192542    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/thanos-sidecar/0.log"
Jul 28 14:07:42.225830 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.225808    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/kube-rbac-proxy-web/0.log"
Jul 28 14:07:42.255993 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.255958    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/kube-rbac-proxy/0.log"
Jul 28 14:07:42.308221 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.308190    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/kube-rbac-proxy-thanos/0.log"
Jul 28 14:07:42.339004 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.338963    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f05cd91a-2eb1-45ba-9840-701f401eb1d5/init-config-reloader/0.log"
Jul 28 14:07:42.621822 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.621797    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5996d84469-snzft_0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc/thanos-query/0.log"
Jul 28 14:07:42.654329 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.654297    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5996d84469-snzft_0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc/kube-rbac-proxy-web/0.log"
Jul 28 14:07:42.695186 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.695159    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5996d84469-snzft_0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc/kube-rbac-proxy/0.log"
Jul 28 14:07:42.728767 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.728737    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5996d84469-snzft_0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc/prom-label-proxy/0.log"
Jul 28 14:07:42.764926 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.764865    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5996d84469-snzft_0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc/kube-rbac-proxy-rules/0.log"
Jul 28 14:07:42.793804 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:42.793781    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5996d84469-snzft_0f8fe314-c9e7-4a9b-b8d6-68b0811d8abc/kube-rbac-proxy-metrics/0.log"
Jul 28 14:07:44.880627 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:44.880602    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54548f458f-9z85b_c49c9874-7c57-4cb2-babc-2ef08dca2bcd/console/0.log"
Jul 28 14:07:46.344743 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.344705    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"]
Jul 28 14:07:46.345117 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.345060    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="copy"
Jul 28 14:07:46.345117 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.345070    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="copy"
Jul 28 14:07:46.345117 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.345082    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="gather"
Jul 28 14:07:46.345117 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.345088    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="gather"
Jul 28 14:07:46.345256 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.345138    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="gather"
Jul 28 14:07:46.345256 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.345145    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84912c46-71ad-45b5-b24a-c0df3e872ffd" containerName="copy"
Jul 28 14:07:46.350189 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.350170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.353066 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.353044    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wskns\"/\"kube-root-ca.crt\""
Jul 28 14:07:46.353175 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.353047    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wskns\"/\"default-dockercfg-b8hlh\""
Jul 28 14:07:46.354263 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.354242    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wskns\"/\"openshift-service-ca.crt\""
Jul 28 14:07:46.357484 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.357462    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"]
Jul 28 14:07:46.457061 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.457025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-lib-modules\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.457061 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.457062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjs9\" (UniqueName: \"kubernetes.io/projected/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-kube-api-access-lcjs9\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.457402 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.457088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-sys\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.457402 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.457109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-podres\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.457402 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.457259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-proc\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.496063 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.496035    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s9nqg_9cdca270-27e6-432b-b86e-2d233993745c/dns/0.log"
Jul 28 14:07:46.521266 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.521244    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-s9nqg_9cdca270-27e6-432b-b86e-2d233993745c/kube-rbac-proxy/0.log"
Jul 28 14:07:46.549507 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.549481    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-js4c9_f9942989-507a-4a45-a294-217549de52ce/dns-node-resolver/0.log"
Jul 28 14:07:46.558351 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-proc\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558469 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-lib-modules\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558469 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjs9\" (UniqueName: \"kubernetes.io/projected/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-kube-api-access-lcjs9\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558469 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558449    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-proc\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558469 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-sys\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558657 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-podres\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558657 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-lib-modules\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558657 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-sys\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.558657 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.558603    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-podres\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.567193 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.567172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjs9\" (UniqueName: \"kubernetes.io/projected/7ebe1bc9-f28a-4058-8bbe-85aecf5dac57-kube-api-access-lcjs9\") pod \"perf-node-gather-daemonset-gmk5v\" (UID: \"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57\") " pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.660953 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.660888    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:46.790668 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:46.790644    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"]
Jul 28 14:07:46.792583 ip-10-0-129-190 kubenswrapper[2576]: W0728 14:07:46.792557    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7ebe1bc9_f28a_4058_8bbe_85aecf5dac57.slice/crio-4d2bf6d4e75b1a3a8074c87ca28c21572c951c398d903d7349bbfa481950d47e WatchSource:0}: Error finding container 4d2bf6d4e75b1a3a8074c87ca28c21572c951c398d903d7349bbfa481950d47e: Status 404 returned error can't find the container with id 4d2bf6d4e75b1a3a8074c87ca28c21572c951c398d903d7349bbfa481950d47e
Jul 28 14:07:47.210534 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:47.210508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kszkc_b54e8191-d84d-430d-9bb1-1c5f76787bfe/node-ca/0.log"
Jul 28 14:07:47.648161 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:47.648127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v" event={"ID":"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57","Type":"ContainerStarted","Data":"1b8a9bdb29828de1f5c0bd44cb8b7bba8b4c7a9b28619fce2b7b05161f0fcf07"}
Jul 28 14:07:47.648566 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:47.648175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v" event={"ID":"7ebe1bc9-f28a-4058-8bbe-85aecf5dac57","Type":"ContainerStarted","Data":"4d2bf6d4e75b1a3a8074c87ca28c21572c951c398d903d7349bbfa481950d47e"}
Jul 28 14:07:47.648566 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:47.648205    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:47.667572 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:47.667524    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v" podStartSLOduration=1.667509454 podStartE2EDuration="1.667509454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:07:47.665129106 +0000 UTC m=+5560.566273499" watchObservedRunningTime="2026-07-28 14:07:47.667509454 +0000 UTC m=+5560.568653844"
Jul 28 14:07:48.520298 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:48.520268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v6s4d_d8b1764d-7728-4e13-99f2-1d17c4d4d896/serve-healthcheck-canary/0.log"
Jul 28 14:07:49.026609 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:49.026583    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kf94z_87172efd-f000-4b9c-8019-f1e930b2a804/kube-rbac-proxy/0.log"
Jul 28 14:07:49.058290 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:49.058266    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kf94z_87172efd-f000-4b9c-8019-f1e930b2a804/exporter/0.log"
Jul 28 14:07:49.093433 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:49.093388    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kf94z_87172efd-f000-4b9c-8019-f1e930b2a804/extractor/0.log"
Jul 28 14:07:51.579159 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:51.579121    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-68bc5447d6-7s6pb_b068c65c-e076-4dfd-9ae4-778197a15297/manager/0.log"
Jul 28 14:07:52.007454 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:52.007358    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-g846t_2cb6ec68-5d6e-4cd3-a231-6399bd271ab7/s3-init/0.log"
Jul 28 14:07:52.025887 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:52.025864    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-g846t_2cb6ec68-5d6e-4cd3-a231-6399bd271ab7/download-hf-model/0.log"
Jul 28 14:07:52.057644 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:52.057623    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-m7mhm_9c70dd70-b158-4f4f-a049-c0ba22f5e2f9/s3-tls-init-custom/0.log"
Jul 28 14:07:52.079003 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:52.078985    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-m7mhm_9c70dd70-b158-4f4f-a049-c0ba22f5e2f9/download-hf-model/0.log"
Jul 28 14:07:52.108422 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:52.108400    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-pbms6_f15e9a40-e555-48bb-ae19-1045f0096d67/s3-tls-init-serving/0.log"
Jul 28 14:07:52.126790 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:52.126772    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-pbms6_f15e9a40-e555-48bb-ae19-1045f0096d67/download-hf-model/0.log"
Jul 28 14:07:53.661988 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:53.661961    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wskns/perf-node-gather-daemonset-gmk5v"
Jul 28 14:07:58.639753 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.639719    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/kube-multus-additional-cni-plugins/0.log"
Jul 28 14:07:58.723185 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.723160    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/egress-router-binary-copy/0.log"
Jul 28 14:07:58.771213 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.771191    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/cni-plugins/0.log"
Jul 28 14:07:58.819286 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.819263    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/bond-cni-plugin/0.log"
Jul 28 14:07:58.867083 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.867065    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/routeoverride-cni/0.log"
Jul 28 14:07:58.919451 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.919398    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/whereabouts-cni-bincopy/0.log"
Jul 28 14:07:58.999345 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:58.999296    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zdksc_01ed658a-d837-4ae6-a286-222f1cda2c28/whereabouts-cni/0.log"
Jul 28 14:07:59.071834 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:59.071812    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ftf2m_0a92a412-e8e8-4181-8d3e-d2388aea297d/kube-multus/0.log"
Jul 28 14:07:59.189900 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:59.189841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hcvxx_c80ac7ce-b3d4-481e-8c0f-6486a5a44827/network-metrics-daemon/0.log"
Jul 28 14:07:59.213971 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:07:59.213950    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hcvxx_c80ac7ce-b3d4-481e-8c0f-6486a5a44827/kube-rbac-proxy/0.log"
Jul 28 14:08:00.558406 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.558358    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/ovn-controller/0.log"
Jul 28 14:08:00.581947 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.581926    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/ovn-acl-logging/0.log"
Jul 28 14:08:00.613094 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.613070    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/kube-rbac-proxy-node/0.log"
Jul 28 14:08:00.637304 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.637277    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 14:08:00.664978 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.664957    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/northd/0.log"
Jul 28 14:08:00.691823 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.691802    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/nbdb/0.log"
Jul 28 14:08:00.719996 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.719972    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/sbdb/0.log"
Jul 28 14:08:00.770454 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:00.770433    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9s78_fd5179df-c0f5-4db1-a387-021f97d1cbd8/ovnkube-controller/0.log"
Jul 28 14:08:01.839029 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:01.838999    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-h9c6h_ce31f5aa-2e18-4923-8c01-a630b25d1e52/network-check-target-container/0.log"
Jul 28 14:08:02.870283 ip-10-0-129-190 kubenswrapper[2576]: I0728 14:08:02.870255    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vqr7r_95bada68-c40f-4006-a489-e937d2475a6d/iptables-alerter/0.log"