Jul 07 02:15:54.948655 ip-10-0-142-238 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 02:15:54.948665 ip-10-0-142-238 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 02:15:54.948672 ip-10-0-142-238 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 02:15:54.948912 ip-10-0-142-238 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 02:16:05.197888 ip-10-0-142-238 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 02:16:05.197904 ip-10-0-142-238 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 71ddba66dd0d49fb8ac1ae10014e4d57 --
Jul 07 02:17:42.605967 ip-10-0-142-238 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 02:17:42.989185 ip-10-0-142-238 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:17:42.989185 ip-10-0-142-238 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 02:17:42.989185 ip-10-0-142-238 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:17:42.989185 ip-10-0-142-238 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 02:17:42.989185 ip-10-0-142-238 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:17:42.990145 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.989374    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 02:17:42.995539 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995513    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:42.995539 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995535    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:42.995539 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995539    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:42.995539 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995542    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:42.995539 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995546    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:42.995539 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995549    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995553    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995556    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995558    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995561    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995563    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995566    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995569    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995571    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995574    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995577    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995580    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995583    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995586    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995588    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995591    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995593    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995595    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995598    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:42.995782 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995600    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995603    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995605    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995607    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995610    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995614    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995618    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995621    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995624    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995627    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995629    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995632    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995635    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995637    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995654    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995658    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995661    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995663    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995666    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995668    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:42.996236 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995671    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995674    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995677    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995680    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995682    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995685    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995688    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995690    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995693    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995695    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995698    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995700    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995703    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995705    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995707    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995710    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995712    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995715    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995717    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995719    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:42.996739 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995723    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995726    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995729    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995732    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995734    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995737    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995739    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995743    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995745    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995748    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995751    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995753    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995756    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995758    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995761    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995764    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995767    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995769    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995772    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:42.997217 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995775    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995778    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995780    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995782    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995785    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995789    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995793    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995796    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995799    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995803    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995807    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995810    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995814    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995817    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995820    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.995823    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996547    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996553    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996555    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:42.997717 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996558    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996561    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996564    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996567    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996569    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996572    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996574    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996577    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996580    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996582    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996586    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996591    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996594    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996597    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996600    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996603    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996605    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996608    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996611    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:42.998167 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996613    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996616    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996618    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996620    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996623    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996626    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996629    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996632    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996634    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996637    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996655    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996658    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996661    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996664    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996667    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996671    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996674    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996677    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996679    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:42.998624 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996682    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996684    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996688    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996691    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996693    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996696    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996699    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996701    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996704    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996706    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996709    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996712    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996714    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996718    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996722    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996724    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996727    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996731    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996734    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:42.999126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996737    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996739    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996743    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996746    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996749    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996752    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996754    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996757    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996760    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996762    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996765    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996767    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996770    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996772    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996775    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996778    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996781    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996783    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996786    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996789    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:42.999590 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996792    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996795    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996798    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996800    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996805    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996807    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996810    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996812    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996815    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996818    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996820    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996823    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996826    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996828    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996832    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996834    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996837    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996839    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.996842    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997271    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:43.000096 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997277    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997280    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997283    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997286    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997289    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997292    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997295    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997298    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997301    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997304    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997306    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997309    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997312    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997315    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997317    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997320    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997324    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997328    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997331    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:43.000595 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997333    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997336    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997339    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997342    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997344    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997347    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997349    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997352    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997355    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997358    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997361    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997363    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997366    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997369    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997371    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997374    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997377    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997379    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997382    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997384    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:43.001070 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997386    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997389    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997391    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997394    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997396    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997398    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997401    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997404    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997406    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997410    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997414    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997417    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997419    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997422    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997425    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997428    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997430    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997433    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997436    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997439    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:43.001549 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997441    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997444    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997447    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997449    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997452    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997455    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997458    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997460    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997463    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997465    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997467    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997470    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997472    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997475    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997477    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997482    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997486    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997489    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997491    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:43.002091 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997501    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997503    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997506    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997508    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997511    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997514    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997516    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997518    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997521    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997523    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997526    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997528    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997531    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997533    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997536    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997539    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997541    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997547    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997549    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.997552    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:43.002535 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997631    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997657    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997664    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997668    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997673    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997676    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997680    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997685    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997688    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997691    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997695    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997698    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997701    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997705    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997708    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997711    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997716    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997719    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997723    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997726    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997729    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997733    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997736    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997743    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 02:17:43.003056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997746    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997749    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997752    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997755    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997758    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997762    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997766    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997769    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997773    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997776    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997778    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997781    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997784    2566 flags.go:64] FLAG: --enable-server="true"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997787    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997792    2566 flags.go:64] FLAG: --event-burst="100"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997795    2566 flags.go:64] FLAG: --event-qps="50"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997798    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997801    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997804    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997808    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997810    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997813    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997817    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997820    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997824    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 02:17:43.003660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997826    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997829    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997832    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997835    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997838    2566 flags.go:64] FLAG: --feature-gates=""
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997842    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997845    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997848    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997851    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997854    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997857    2566 flags.go:64] FLAG: --help="false"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997859    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997864    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997867    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997870    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997877    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997881    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997884    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997887    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997890    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997893    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997895    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997899    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997902    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 07 02:17:43.004343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997905    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997907    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997910    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997913    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997916    2566 flags.go:64] FLAG: --lock-file=""
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997919    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997923    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997927    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997933    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997936    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997939    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997941    2566 flags.go:64] FLAG: --logging-format="text"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997944    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997947    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997950    2566 flags.go:64] FLAG: --manifest-url=""
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997953    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997957    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997960    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997964    2566 flags.go:64] FLAG: --max-pods="110"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997967    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997972    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997975    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997978    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997981    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997984    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 02:17:43.005034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997987    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997995    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.997998    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998001    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998004    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998007    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998013    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998015    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998018    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998021    2566 flags.go:64] FLAG: --port="10250"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998024    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998027    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f4f35be2e1037780"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998031    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998033    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998037    2566 flags.go:64] FLAG: --register-node="true"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998041    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998045    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998047    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998050    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998053    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998057    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998060    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998062    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998065    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998068    2566 flags.go:64] FLAG: --runonce="false"
Jul 07 02:17:43.005665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998071    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998074    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998078    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998081    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998084    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998086    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998090    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998093    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998095    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998098    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998101    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998104    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998107    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998110    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998112    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998118    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998121    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998123    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998127    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998130    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998133    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998136    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998140    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998143    2566 flags.go:64] FLAG: --v="2"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998147    2566 flags.go:64] FLAG: --version="false"
Jul 07 02:17:43.006284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998151    2566 flags.go:64] FLAG: --vmodule=""
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998155    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.998158    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998275    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998278    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998281    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998284    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998287    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998289    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998293    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998296    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998298    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998301    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998303    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998306    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998308    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998311    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998314    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998317    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998319    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:43.006947 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998321    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998324    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998328    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998331    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998334    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998337    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998340    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998342    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998346    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998350    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998354    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998357    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998360    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998364    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998367    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998370    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998373    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998376    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:43.007465 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998378    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998381    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998384    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998388    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998390    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998393    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998395    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998398    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998401    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998403    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998406    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998408    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998411    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998415    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998418    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998421    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998424    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998427    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998430    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:43.007929 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998433    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998435    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998438    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998440    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998445    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998447    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998450    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998453    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998455    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998458    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998460    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998462    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998465    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998468    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998471    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998474    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998478    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998480    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998483    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998486    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:43.008382 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998488    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998490    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998493    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998496    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998498    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998501    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998503    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998505    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998508    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998511    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998513    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998516    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998518    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998521    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998523    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998525    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998529    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998532    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998534    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998538    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:43.008881 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998540    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998543    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998545    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998548    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:42.998550    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:42.999075    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006045    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006062    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006079    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006085    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006143    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006149    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006154    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006157    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:43.009400 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006161    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006165    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006168    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006171    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006174    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006177    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006180    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006183    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006186    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006189    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006191    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006194    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006196    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006199    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006201    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006204    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006206    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006209    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006213    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:43.009763 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006216    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006219    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006222    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006225    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006227    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006229    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006234    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006237    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006239    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006242    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006245    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006247    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006250    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006252    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006256    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006258    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006261    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006263    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006266    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006269    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:43.010222 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006271    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006274    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006276    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006279    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006282    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006284    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006287    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006290    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006293    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006296    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006299    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006302    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006304    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006307    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006310    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006313    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006315    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006317    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006322    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006325    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:43.010857 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006327    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006330    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006333    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006335    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006337    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006340    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006343    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006345    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006348    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006350    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006352    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006355    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006357    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006360    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006362    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006365    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006367    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006370    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006372    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006374    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:43.011339 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006377    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006379    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006382    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006384    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006387    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006389    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006392    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006394    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006397    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006400    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006403    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006406    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006409    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006412    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006414    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:43.011862 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006417    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006422    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006538    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006543    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006547    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006550    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006553    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006555    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006558    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006560    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006563    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006565    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006568    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006570    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006573    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:43.012397 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006576    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006578    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006580    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006583    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006586    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006588    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006591    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006594    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006596    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006599    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006601    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006603    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006606    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006609    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006612    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006615    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006618    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006620    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006623    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:43.013239 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006625    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006628    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006630    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006633    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006635    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006652    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006657    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006661    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006663    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006666    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006669    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006672    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006674    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006677    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006679    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006682    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006684    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006687    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006689    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006692    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:43.013733 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006694    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006698    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006701    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006704    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006707    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006709    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006712    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006715    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006717    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006722    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006726    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006730    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006732    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006735    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006738    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006740    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006743    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006745    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006748    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:43.014391 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006750    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006753    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006755    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006758    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006760    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006763    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006765    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006768    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006770    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006773    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006775    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006778    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006780    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006783    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006785    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006788    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006790    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006793    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006796    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006799    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:43.014974 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006801    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006804    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006807    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006819    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006823    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006826    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006828    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:43.006832    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006838    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.006968    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.009114    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.010398    2566 server.go:1019] "Starting client certificate rotation"
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.010511    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:17:43.015452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.010561    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:17:43.015807 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.012857    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:17:43.015807 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.013983    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:17:43.025567 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.025544    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 07 02:17:43.032929 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.032904    2566 log.go:25] "Validated CRI v1 image API"
Jul 07 02:17:43.035041 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.035018    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 02:17:43.038116 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.038095    2566 fs.go:135] Filesystem UUIDs: map[012e3c84-e9ad-4c72-83d5-83adca1d1bca:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c14557bd-5c66-4578-aefb-2fad403aedf8:/dev/nvme0n1p4]
Jul 07 02:17:43.038184 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.038116    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 02:17:43.043200 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.043177    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:17:43.044791 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.044667    2566 manager.go:216] Machine: {Timestamp:2026-07-07 02:17:43.042477555 +0000 UTC m=+0.323616097 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100708 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2546ed606be4bdf822221b12fdcafa SystemUUID:ec2546ed-606b-e4bd-f822-221b12fdcafa BootID:71ddba66-dd0d-49fb-8ac1-ae10014e4d57 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a7:e4:9c:bd:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a7:e4:9c:bd:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:bc:a8:6c:4b:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 02:17:43.044791 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.044789    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 02:17:43.044907 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.044891    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 02:17:43.048323 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.048288    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 02:17:43.048510 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.048325    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 02:17:43.049146 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.049121    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 02:17:43.049146 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.049151    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 02:17:43.049243 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.049163    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:17:43.050386 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.050371    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:17:43.051804 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.051790    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 02:17:43.054822 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.054795    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:17:43.054977 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.054965    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 02:17:43.055033 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055016    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 02:17:43.055079 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055038    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 02:17:43.055079 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055061    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 02:17:43.055079 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055075    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 07 02:17:43.055191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055089    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 02:17:43.055523 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055508    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:17:43.055660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.055531    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:17:43.057826 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.057800    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 02:17:43.059817 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.059800    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 02:17:43.059867 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.059846    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 02:17:43.061021 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061008    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061025    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061031    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061037    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061045    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061052    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061060    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061067    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061074    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 02:17:43.061084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061080    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 02:17:43.061321 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061093    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 02:17:43.061321 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061099    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 02:17:43.061854 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061844    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 02:17:43.061854 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.061854    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 02:17:43.065575 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.065549    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 02:17:43.065681 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.065572    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 02:17:43.065681 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.065677    2566 server.go:1268] "Started kubelet"
Jul 07 02:17:43.065775 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.065678    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-238.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 02:17:43.065830 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.065754    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 02:17:43.066061 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.065890    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 02:17:43.066180 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.066162    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 02:17:43.066180 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.066170    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgrsm"
Jul 07 02:17:43.066678 ip-10-0-142-238 systemd[1]: Started Kubernetes Kubelet.
Jul 07 02:17:43.071782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.071759    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 02:17:43.075048 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.075020    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qgrsm"
Jul 07 02:17:43.075156 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.075138    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 02:17:43.075434 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.075329    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 02:17:43.080180 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.080162    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 02:17:43.080308 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.080191    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 02:17:43.080308 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.080233    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.080984 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.080966    2566 factory.go:55] Registering systemd factory
Jul 07 02:17:43.080984 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.080984    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 07 02:17:43.081237 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081223    2566 factory.go:153] Registering CRI-O factory
Jul 07 02:17:43.081304 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081240    2566 factory.go:223] Registration of the crio container factory successfully
Jul 07 02:17:43.081304 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081277    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 02:17:43.081304 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081295    2566 factory.go:103] Registering Raw factory
Jul 07 02:17:43.081304 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081305    2566 manager.go:1196] Started watching for new ooms in manager
Jul 07 02:17:43.081479 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081461    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 02:17:43.081529 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081460    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 02:17:43.081529 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081492    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 02:17:43.081671 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081625    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 02:17:43.081671 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081657    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 02:17:43.081862 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.081851    2566 manager.go:349] Starting recovery of all containers
Jul 07 02:17:43.081892 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.080805    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-238.ec2.internal.18bfe05bf7c67c12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-238.ec2.internal,UID:ip-10-0-142-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-238.ec2.internal,},FirstTimestamp:2026-07-07 02:17:43.065631762 +0000 UTC m=+0.346770300,LastTimestamp:2026-07-07 02:17:43.065631762 +0000 UTC m=+0.346770300,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-238.ec2.internal,}"
Jul 07 02:17:43.086420 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.086010    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:43.092619 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.089654    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-238.ec2.internal\" not found" node="ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.093256 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.092835    2566 manager.go:354] Recovery completed
Jul 07 02:17:43.097714 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.097696    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.099856 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.099837    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.099924 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.099871    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.099924 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.099893    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.101506 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.101461    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 02:17:43.101506 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.101477    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 02:17:43.101506 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.101500    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:17:43.101739 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.101487    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.102590 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.102573    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.102683 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.102603    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.102683 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.102613    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.102875 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.102862    2566 policy_none.go:49] "None policy: Start"
Jul 07 02:17:43.102917 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.102886    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 02:17:43.102917 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.102898    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 02:17:43.103943 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.103927    2566 policy_none.go:47] "Start"
Jul 07 02:17:43.154744 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.154507    2566 manager.go:337] "Starting Device Plugin manager"
Jul 07 02:17:43.154945 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.154774    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 02:17:43.154945 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.154789    2566 server.go:87] "Starting device plugin registration server"
Jul 07 02:17:43.155122 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.155109    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 02:17:43.155181 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.155124    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 02:17:43.155227 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.155214    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 02:17:43.155321 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.155305    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 02:17:43.155397 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.155321    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 02:17:43.155916 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.155885    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 02:17:43.156027 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.155937    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.218782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.218745    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 02:17:43.219949 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.219927    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 02:17:43.220069 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.219973    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 02:17:43.220069 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.220004    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 02:17:43.220069 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.220048    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 02:17:43.222734 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.222712    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:43.256189 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.256105    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.257708 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.257686    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.257814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.257723    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.257814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.257736    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.257814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.257759    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.264362 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.264342    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.264448 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.264371    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-238.ec2.internal\": node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.281025 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.280993    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.320945 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.320897    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"]
Jul 07 02:17:43.321075 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.320996    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.322028 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.322011    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.322130 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.322040    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.322130 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.322051    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.323484 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.323470    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.323626 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.323611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.323684 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.323657    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.324284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.324261    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.324363 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.324292    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.324363 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.324302    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.324363 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.324266    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.324468 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.324368    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.324468 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.324378    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.325604 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.325587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.325669 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.325624    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:43.326338 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.326320    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:43.326422 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.326344    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:43.326422 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.326353    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:43.351581 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.351553    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-238.ec2.internal\" not found" node="ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.356450 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.356429    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-238.ec2.internal\" not found" node="ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.381494 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.381469    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.383719 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.383698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0ca3d7c0f62b242a594ca6f1f035eef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal\" (UID: \"d0ca3d7c0f62b242a594ca6f1f035eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.383786 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.383728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ca3d7c0f62b242a594ca6f1f035eef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal\" (UID: \"d0ca3d7c0f62b242a594ca6f1f035eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.383786 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.383748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b0b802c6511aef6dcda6b6d308a1cf9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.ec2.internal\" (UID: \"4b0b802c6511aef6dcda6b6d308a1cf9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.481727 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.481684    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.484905 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.484883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0ca3d7c0f62b242a594ca6f1f035eef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal\" (UID: \"d0ca3d7c0f62b242a594ca6f1f035eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.484985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.484914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ca3d7c0f62b242a594ca6f1f035eef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal\" (UID: \"d0ca3d7c0f62b242a594ca6f1f035eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.484985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.484935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b0b802c6511aef6dcda6b6d308a1cf9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.ec2.internal\" (UID: \"4b0b802c6511aef6dcda6b6d308a1cf9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.484985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.484976    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b0b802c6511aef6dcda6b6d308a1cf9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-238.ec2.internal\" (UID: \"4b0b802c6511aef6dcda6b6d308a1cf9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.485109 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.484989    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0ca3d7c0f62b242a594ca6f1f035eef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal\" (UID: \"d0ca3d7c0f62b242a594ca6f1f035eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.485109 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.485012    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ca3d7c0f62b242a594ca6f1f035eef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal\" (UID: \"d0ca3d7c0f62b242a594ca6f1f035eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.582541 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.582456    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:43.654837 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.654806    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.659926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:43.659905    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:43.682991 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:43.682961    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:45.405582 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:45.405534    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:45.506157 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:45.506092    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:45.606763 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:45.606717    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:45.632052 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.632031    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 02:17:45.632455 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.632196    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 02:17:45.632455 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.632206    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 02:17:45.704295 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.704244    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 02:12:44 +0000 UTC" deadline="2027-12-24 04:21:49.236249347 +0000 UTC"
Jul 07 02:17:45.704295 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.704277    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12842h4m3.531974914s"
Jul 07 02:17:45.707810 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:45.707788    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-238.ec2.internal\" not found"
Jul 07 02:17:45.734980 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.734952    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:45.802931 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.802894    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:45.820992 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.820965    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:17:45.822653 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.822636    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal"
Jul 07 02:17:45.849627 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.849609    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:17:45.854900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.854877    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:45.858747 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:45.858720    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0b802c6511aef6dcda6b6d308a1cf9.slice/crio-f852b623d3ef0bfc1d2b6df4cb98717dcbf32632f8164b5a678132541a83acd1 WatchSource:0}: Error finding container f852b623d3ef0bfc1d2b6df4cb98717dcbf32632f8164b5a678132541a83acd1: Status 404 returned error can't find the container with id f852b623d3ef0bfc1d2b6df4cb98717dcbf32632f8164b5a678132541a83acd1
Jul 07 02:17:45.859186 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:45.859169    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ca3d7c0f62b242a594ca6f1f035eef.slice/crio-03ec6e93934826fe18295385dd8f594043da350c554ce67ecdc3de550a67b153 WatchSource:0}: Error finding container 03ec6e93934826fe18295385dd8f594043da350c554ce67ecdc3de550a67b153: Status 404 returned error can't find the container with id 03ec6e93934826fe18295385dd8f594043da350c554ce67ecdc3de550a67b153
Jul 07 02:17:45.863969 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:45.863918    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:17:46.677643 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.677615    2566 apiserver.go:52] "Watching apiserver"
Jul 07 02:17:46.687216 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.687184    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 02:17:46.687584 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.687556    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-gvjbd","kube-system/konnectivity-agent-lv8qk","kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal","openshift-image-registry/node-ca-jvhhh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal","openshift-multus/network-metrics-daemon-v2khl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw","openshift-cluster-node-tuning-operator/tuned-7gkcn","openshift-dns/node-resolver-r9mtv","openshift-multus/multus-9vrf7","openshift-multus/multus-additional-cni-plugins-r2x6b","openshift-network-diagnostics/network-check-target-ns6nx","openshift-network-operator/iptables-alerter-phrvp","openshift-ovn-kubernetes/ovnkube-node-g4dxr"]
Jul 07 02:17:46.689074 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.689054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.690195 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.690173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.692213 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.691987    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.692213 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.692213 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.692213 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692059    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vbpwp\""
Jul 07 02:17:46.692449 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692345    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 02:17:46.692845 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tlcsx\""
Jul 07 02:17:46.692845 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692702    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 02:17:46.692845 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.692697    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 02:17:46.693063 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.693048    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.694232 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.694214    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:46.694318 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.694283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:46.694618 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.694603    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.694750 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.694655    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h2724\""
Jul 07 02:17:46.694840 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.694784    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.695465 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.695231    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.695465 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.695428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:46.695592 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.695483    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:46.695592 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.695550    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.695592 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.695587    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bkggk\""
Jul 07 02:17:46.695758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.695550    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 02:17:46.696891 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.696872    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.698763 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.698746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.699241 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.699224    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 02:17:46.699523 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.699504    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 02:17:46.699612 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.699601    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 02:17:46.699756 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.699741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.699915 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.699895    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.699990 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.699929    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cb4gb\""
Jul 07 02:17:46.701004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.700985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.701577 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.701556    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.702191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.702176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.702852 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.702836    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 02:17:46.702940 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.702917    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.703467 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.703442    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qp6jm\""
Jul 07 02:17:46.703553 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.703502    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.703553 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.703540    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 02:17:46.703553 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.703549    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 02:17:46.703888 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.703560    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:17:46.703949 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.703921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-klqwd\""
Jul 07 02:17:46.705042 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.704933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.705413 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.705383    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.705413 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.705410    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.705562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.705521    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-72qgq\""
Jul 07 02:17:46.707529 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.707513    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 02:17:46.707622 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.707518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.708264 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.708202    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.708627 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.708604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 02:17:46.709607 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.708783    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 02:17:46.709607 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.708950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lmft8\""
Jul 07 02:17:46.709787 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.709751    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 02:17:46.714926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.714908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-ovnkube-script-lib\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715018 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.714934    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-lib-modules\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.715018 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.714960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysconfig\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.715018 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.714982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysctl-conf\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.715018 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-sys\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-tuned\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-cnibin\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-registration-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-conf-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-etc-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.715220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3cee6f8-980e-4530-b615-b02786a8e117-kubelet-config\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-systemd-units\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-slash\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-run-netns\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-host\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbz72\" (UniqueName: \"kubernetes.io/projected/cc440849-966b-4c47-8e20-36454af21e2c-kube-api-access-lbz72\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvpb\" (UniqueName: \"kubernetes.io/projected/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-kube-api-access-8rvpb\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fd8z\" (UniqueName: \"kubernetes.io/projected/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-kube-api-access-8fd8z\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqgp\" (UniqueName: \"kubernetes.io/projected/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-kube-api-access-mgqgp\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-var-lib-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-cni-netd\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-env-overrides\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.715538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622rq\" (UniqueName: \"kubernetes.io/projected/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-kube-api-access-622rq\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-cni-bin\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-kubelet\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sddl\" (UniqueName: \"kubernetes.io/projected/327320b3-4c10-453f-811b-48f5033b92ce-kube-api-access-7sddl\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bc7941d-4a32-488e-9c02-cb43da0a5d04-tmp\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq9f\" (UniqueName: \"kubernetes.io/projected/9bc7941d-4a32-488e-9c02-cb43da0a5d04-kube-api-access-swq9f\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-daemon-config\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-cni-bin\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-kubernetes\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysctl-d\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715888    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-os-release\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-cni-multus\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-etc-kubernetes\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.715951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.716078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8096f-0d4c-45c5-a548-316323d7dc8a-konnectivity-ca\") pod \"konnectivity-agent-lv8qk\" (UID: \"d0a8096f-0d4c-45c5-a548-316323d7dc8a\") " pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-var-lib-kubelet\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc440849-966b-4c47-8e20-36454af21e2c-serviceca\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-hosts-file\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-tmp-dir\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3cee6f8-980e-4530-b615-b02786a8e117-dbus\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/327320b3-4c10-453f-811b-48f5033b92ce-ovn-node-metrics-cert\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxlz\" (UniqueName: \"kubernetes.io/projected/cc8b27f6-e10e-4e0f-b760-9940e55c4416-kube-api-access-nnxlz\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-system-cni-dir\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-os-release\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-k8s-cni-cncf-io\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-node-log\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-log-socket\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-ovnkube-config\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-run\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.716737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716414    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc440849-966b-4c47-8e20-36454af21e2c-host\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716494    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-systemd\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0a8096f-0d4c-45c5-a548-316323d7dc8a-agent-certs\") pod \"konnectivity-agent-lv8qk\" (UID: \"d0a8096f-0d4c-45c5-a548-316323d7dc8a\") " pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-modprobe-d\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqh5\" (UniqueName: \"kubernetes.io/projected/eb530956-7b82-4d88-975f-3c3e042b7fb8-kube-api-access-xfqh5\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-netns\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-socket-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-iptables-alerter-script\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-kubelet\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-ovn\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:46.717430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-device-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-sys-fs\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.716996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-host-slash\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cnibin\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-system-cni-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-cni-binary-copy\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-hostroot\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-multus-certs\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-systemd\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-cni-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-socket-dir-parent\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.718178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.717563    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:17:46.747493 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.747468    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6q5c"
Jul 07 02:17:46.753383 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.753360    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6q5c"
Jul 07 02:17:46.804245 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.804219    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 02:17:46.818504 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-host\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.818661 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbz72\" (UniqueName: \"kubernetes.io/projected/cc440849-966b-4c47-8e20-36454af21e2c-kube-api-access-lbz72\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.818661 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvpb\" (UniqueName: \"kubernetes.io/projected/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-kube-api-access-8rvpb\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.818661 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fd8z\" (UniqueName: \"kubernetes.io/projected/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-kube-api-access-8fd8z\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.818661 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818584    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-host\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.818661 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqgp\" (UniqueName: \"kubernetes.io/projected/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-kube-api-access-mgqgp\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-var-lib-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-cni-netd\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-env-overrides\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622rq\" (UniqueName: \"kubernetes.io/projected/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-kube-api-access-622rq\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818830    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-var-lib-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818842    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-cni-netd\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.818934 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-cni-bin\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-kubelet\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818959    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-cni-bin\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.818971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819000    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-kubelet\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819027    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sddl\" (UniqueName: \"kubernetes.io/projected/327320b3-4c10-453f-811b-48f5033b92ce-kube-api-access-7sddl\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bc7941d-4a32-488e-9c02-cb43da0a5d04-tmp\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq9f\" (UniqueName: \"kubernetes.io/projected/9bc7941d-4a32-488e-9c02-cb43da0a5d04-kube-api-access-swq9f\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-daemon-config\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-cni-bin\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-kubernetes\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysctl-d\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-os-release\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-cni-multus\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.819284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819289    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-etc-kubernetes\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8096f-0d4c-45c5-a548-316323d7dc8a-konnectivity-ca\") pod \"konnectivity-agent-lv8qk\" (UID: \"d0a8096f-0d4c-45c5-a548-316323d7dc8a\") " pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819379    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-env-overrides\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-var-lib-kubelet\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc440849-966b-4c47-8e20-36454af21e2c-serviceca\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-hosts-file\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819447    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-cni-bin\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-tmp-dir\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819477    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-os-release\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819564    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-kubernetes\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysctl-d\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819632    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-var-lib-cni-multus\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819691    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3cee6f8-980e-4530-b615-b02786a8e117-dbus\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819700    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-etc-kubernetes\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819749    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-var-lib-kubelet\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.820031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819749    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819792    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-hosts-file\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3cee6f8-980e-4530-b615-b02786a8e117-dbus\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/327320b3-4c10-453f-811b-48f5033b92ce-ovn-node-metrics-cert\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.819993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxlz\" (UniqueName: \"kubernetes.io/projected/cc8b27f6-e10e-4e0f-b760-9940e55c4416-kube-api-access-nnxlz\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-system-cni-dir\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-os-release\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820063    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-tmp-dir\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820069    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-daemon-config\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-k8s-cni-cncf-io\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820124    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-system-cni-dir\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-node-log\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820154    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-os-release\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820164    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d0a8096f-0d4c-45c5-a548-316323d7dc8a-konnectivity-ca\") pod \"konnectivity-agent-lv8qk\" (UID: \"d0a8096f-0d4c-45c5-a548-316323d7dc8a\") " pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820175    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-log-socket\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820210    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cc440849-966b-4c47-8e20-36454af21e2c-serviceca\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.820871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-ovnkube-config\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820230    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-log-socket\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-run\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820260    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-k8s-cni-cncf-io\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc440849-966b-4c47-8e20-36454af21e2c-host\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820281    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-run\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820300    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820325    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cc440849-966b-4c47-8e20-36454af21e2c-host\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-systemd\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820425    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-systemd\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0a8096f-0d4c-45c5-a548-316323d7dc8a-agent-certs\") pod \"konnectivity-agent-lv8qk\" (UID: \"d0a8096f-0d4c-45c5-a548-316323d7dc8a\") " pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-modprobe-d\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.821526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820474    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-run-ovn-kubernetes\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqh5\" (UniqueName: \"kubernetes.io/projected/eb530956-7b82-4d88-975f-3c3e042b7fb8-kube-api-access-xfqh5\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820800    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-modprobe-d\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820801    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-ovnkube-config\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820854    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-node-log\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-netns\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-socket-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820890    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-netns\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821016    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821011    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-socket-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-iptables-alerter-script\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-kubelet\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-ovn\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-device-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-sys-fs\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-host-slash\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.822046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cnibin\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-system-cni-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-cni-binary-copy\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-hostroot\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821380    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-iptables-alerter-script\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.820953    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821436    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-kubelet\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821452    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-sys-fs\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821475    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-ovn\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821505    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-system-cni-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821547    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-host-slash\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.821561    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821582    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cnibin\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.821642    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:47.321608597 +0000 UTC m=+2.980717363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-multus-certs\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-systemd\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-cni-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.822848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-socket-dir-parent\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821849    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-device-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-ovnkube-script-lib\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-lib-modules\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.821924    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821961    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-hostroot\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.821976    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:47.321961312 +0000 UTC m=+2.981070058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.821993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysconfig\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822006    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-cni-binary-copy\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysctl-conf\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822008    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-host-run-multus-certs\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822043    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-run-systemd\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-sys\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822123    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-lib-modules\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822171    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysctl-conf\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822175    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-sysconfig\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.823606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-tuned\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822221    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-socket-dir-parent\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822283    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-cni-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-cnibin\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-registration-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-conf-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-etc-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822394    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3cee6f8-980e-4530-b615-b02786a8e117-kubelet-config\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-systemd-units\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-slash\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822515    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-multus-conf-dir\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-run-netns\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822536    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/327320b3-4c10-453f-811b-48f5033b92ce-ovnkube-script-lib\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822556    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-run-netns\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822606    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-host-slash\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822625    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bc7941d-4a32-488e-9c02-cb43da0a5d04-sys\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.824322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822625    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3cee6f8-980e-4530-b615-b02786a8e117-kubelet-config\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822638    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-systemd-units\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc8b27f6-e10e-4e0f-b760-9940e55c4416-registration-dir\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822687    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/327320b3-4c10-453f-811b-48f5033b92ce-etc-openvswitch\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822781    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-cnibin\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.822794    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.824403    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/327320b3-4c10-453f-811b-48f5033b92ce-ovn-node-metrics-cert\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.824456    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d0a8096f-0d4c-45c5-a548-316323d7dc8a-agent-certs\") pod \"konnectivity-agent-lv8qk\" (UID: \"d0a8096f-0d4c-45c5-a548-316323d7dc8a\") " pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:46.825076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.824599    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bc7941d-4a32-488e-9c02-cb43da0a5d04-tmp\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.825887 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.825866    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9bc7941d-4a32-488e-9c02-cb43da0a5d04-etc-tuned\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.828748 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.828102    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqgp\" (UniqueName: \"kubernetes.io/projected/a1868bb6-d73c-4ac4-b26a-cd9e60b97515-kube-api-access-mgqgp\") pod \"node-resolver-r9mtv\" (UID: \"a1868bb6-d73c-4ac4-b26a-cd9e60b97515\") " pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:46.828748 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.828554    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbz72\" (UniqueName: \"kubernetes.io/projected/cc440849-966b-4c47-8e20-36454af21e2c-kube-api-access-lbz72\") pod \"node-ca-jvhhh\" (UID: \"cc440849-966b-4c47-8e20-36454af21e2c\") " pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:46.829772 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.829322    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:46.829772 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.829348    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:46.829772 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.829362    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:46.829772 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.829546    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq9f\" (UniqueName: \"kubernetes.io/projected/9bc7941d-4a32-488e-9c02-cb43da0a5d04-kube-api-access-swq9f\") pod \"tuned-7gkcn\" (UID: \"9bc7941d-4a32-488e-9c02-cb43da0a5d04\") " pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:46.830007 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:46.829438    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:17:47.329419013 +0000 UTC m=+2.988527774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:46.832207 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.831441    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxlz\" (UniqueName: \"kubernetes.io/projected/cc8b27f6-e10e-4e0f-b760-9940e55c4416-kube-api-access-nnxlz\") pod \"aws-ebs-csi-driver-node-wn4fw\" (UID: \"cc8b27f6-e10e-4e0f-b760-9940e55c4416\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:46.833094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.833031    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-622rq\" (UniqueName: \"kubernetes.io/projected/c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e-kube-api-access-622rq\") pod \"multus-additional-cni-plugins-r2x6b\" (UID: \"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e\") " pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:46.833192 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.833090    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fd8z\" (UniqueName: \"kubernetes.io/projected/af5d8a33-1406-4710-a9ed-3ce2b9e8788b-kube-api-access-8fd8z\") pod \"iptables-alerter-phrvp\" (UID: \"af5d8a33-1406-4710-a9ed-3ce2b9e8788b\") " pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:46.834366 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.834342    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvpb\" (UniqueName: \"kubernetes.io/projected/2054e60e-8de7-4424-a6fc-bccb5d0fe9c6-kube-api-access-8rvpb\") pod \"multus-9vrf7\" (UID: \"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6\") " pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:46.835039 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.835015    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sddl\" (UniqueName: \"kubernetes.io/projected/327320b3-4c10-453f-811b-48f5033b92ce-kube-api-access-7sddl\") pod \"ovnkube-node-g4dxr\" (UID: \"327320b3-4c10-453f-811b-48f5033b92ce\") " pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:46.835130 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.835113    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqh5\" (UniqueName: \"kubernetes.io/projected/eb530956-7b82-4d88-975f-3c3e042b7fb8-kube-api-access-xfqh5\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:46.846480 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.846428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal" event={"ID":"d0ca3d7c0f62b242a594ca6f1f035eef","Type":"ContainerStarted","Data":"03ec6e93934826fe18295385dd8f594043da350c554ce67ecdc3de550a67b153"}
Jul 07 02:17:46.847603 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:46.847576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal" event={"ID":"4b0b802c6511aef6dcda6b6d308a1cf9","Type":"ContainerStarted","Data":"f852b623d3ef0bfc1d2b6df4cb98717dcbf32632f8164b5a678132541a83acd1"}
Jul 07 02:17:47.001616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.001533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw"
Jul 07 02:17:47.009807 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.009782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:17:47.018647 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.018625    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7gkcn"
Jul 07 02:17:47.024394 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.024371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvhhh"
Jul 07 02:17:47.030778 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.030757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2x6b"
Jul 07 02:17:47.037332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.037312    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phrvp"
Jul 07 02:17:47.053017 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.052978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9vrf7"
Jul 07 02:17:47.059976 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.059940    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r9mtv"
Jul 07 02:17:47.066830 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.066806    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:17:47.087044 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.087015    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:47.109133 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.109101    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:47.326444 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.326356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:47.326444 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.326422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:47.326662 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.326511    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:47.326662 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.326527    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:47.326662 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.326562    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.326549213 +0000 UTC m=+3.985657957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:47.326662 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.326587    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.326570032 +0000 UTC m=+3.985678772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:47.427429 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.427379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:47.427617 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.427591    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:47.427690 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.427620    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:47.427690 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.427634    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:47.427772 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:47.427719    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.427699473 +0000 UTC m=+4.086808236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:47.756837 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.756742    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:46 +0000 UTC" deadline="2027-12-02 12:34:47.2566313 +0000 UTC"
Jul 07 02:17:47.756837 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.756779    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12322h16m59.499856534s"
Jul 07 02:17:47.842516 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.842489    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327320b3_4c10_453f_811b_48f5033b92ce.slice/crio-ab7ec745984857c917d13eaceaaababf0cc91889580e31c87f403e6f0bc844ba WatchSource:0}: Error finding container ab7ec745984857c917d13eaceaaababf0cc91889580e31c87f403e6f0bc844ba: Status 404 returned error can't find the container with id ab7ec745984857c917d13eaceaaababf0cc91889580e31c87f403e6f0bc844ba
Jul 07 02:17:47.844835 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.844811    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc440849_966b_4c47_8e20_36454af21e2c.slice/crio-747c9819928ae51de73aa2ac602949a8145075100d4370c47011cbff38ae7cbb WatchSource:0}: Error finding container 747c9819928ae51de73aa2ac602949a8145075100d4370c47011cbff38ae7cbb: Status 404 returned error can't find the container with id 747c9819928ae51de73aa2ac602949a8145075100d4370c47011cbff38ae7cbb
Jul 07 02:17:47.849645 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.849610    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1868bb6_d73c_4ac4_b26a_cd9e60b97515.slice/crio-4c0a98c23a64bbd8eb4f111286b619ebcf613c60459f98a7732868a6e572942f WatchSource:0}: Error finding container 4c0a98c23a64bbd8eb4f111286b619ebcf613c60459f98a7732868a6e572942f: Status 404 returned error can't find the container with id 4c0a98c23a64bbd8eb4f111286b619ebcf613c60459f98a7732868a6e572942f
Jul 07 02:17:47.850761 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.850304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvhhh" event={"ID":"cc440849-966b-4c47-8e20-36454af21e2c","Type":"ContainerStarted","Data":"747c9819928ae51de73aa2ac602949a8145075100d4370c47011cbff38ae7cbb"}
Jul 07 02:17:47.851570 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.851546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8b27f6_e10e_4e0f_b760_9940e55c4416.slice/crio-d056c6339320fe332347b33524fd513caa391fc7dcbe89aa92af55b95d0314fa WatchSource:0}: Error finding container d056c6339320fe332347b33524fd513caa391fc7dcbe89aa92af55b95d0314fa: Status 404 returned error can't find the container with id d056c6339320fe332347b33524fd513caa391fc7dcbe89aa92af55b95d0314fa
Jul 07 02:17:47.852323 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:47.852300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"ab7ec745984857c917d13eaceaaababf0cc91889580e31c87f403e6f0bc844ba"}
Jul 07 02:17:47.853475 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.853454    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2054e60e_8de7_4424_a6fc_bccb5d0fe9c6.slice/crio-4b0c049be5f53d76eeefe49430d66bebea40e98c526df51ae6d1f1f58fcd57ad WatchSource:0}: Error finding container 4b0c049be5f53d76eeefe49430d66bebea40e98c526df51ae6d1f1f58fcd57ad: Status 404 returned error can't find the container with id 4b0c049be5f53d76eeefe49430d66bebea40e98c526df51ae6d1f1f58fcd57ad
Jul 07 02:17:47.854840 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.854817    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc7941d_4a32_488e_9c02_cb43da0a5d04.slice/crio-5ab31b3e923997305455ed058e45661e2d96fcee92e68c2e71180edd48f3fe64 WatchSource:0}: Error finding container 5ab31b3e923997305455ed058e45661e2d96fcee92e68c2e71180edd48f3fe64: Status 404 returned error can't find the container with id 5ab31b3e923997305455ed058e45661e2d96fcee92e68c2e71180edd48f3fe64
Jul 07 02:17:47.855931 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.855897    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc245e94b_37ed_45d9_a4aa_b2ddc1e77b6e.slice/crio-eca75dbaccac4713e0c731e505504414d22f7ea17c07924a1f80cb366c45db43 WatchSource:0}: Error finding container eca75dbaccac4713e0c731e505504414d22f7ea17c07924a1f80cb366c45db43: Status 404 returned error can't find the container with id eca75dbaccac4713e0c731e505504414d22f7ea17c07924a1f80cb366c45db43
Jul 07 02:17:47.856570 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.856550    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5d8a33_1406_4710_a9ed_3ce2b9e8788b.slice/crio-0b124d0fb5b47277d5be32e66ec40b7ec62d7667dfa9a67961c463efcb678486 WatchSource:0}: Error finding container 0b124d0fb5b47277d5be32e66ec40b7ec62d7667dfa9a67961c463efcb678486: Status 404 returned error can't find the container with id 0b124d0fb5b47277d5be32e66ec40b7ec62d7667dfa9a67961c463efcb678486
Jul 07 02:17:47.861370 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:17:47.861351    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a8096f_0d4c_45c5_a548_316323d7dc8a.slice/crio-d1231cd63cb42a021ba00205ffa797b14c1b1b826c995f8525987540a31791ad WatchSource:0}: Error finding container d1231cd63cb42a021ba00205ffa797b14c1b1b826c995f8525987540a31791ad: Status 404 returned error can't find the container with id d1231cd63cb42a021ba00205ffa797b14c1b1b826c995f8525987540a31791ad
Jul 07 02:17:48.334698 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.334380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:48.334698 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.334661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:48.334906 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.334735    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:48.334906 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.334806    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:48.334906 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.334815    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:50.334792988 +0000 UTC m=+5.993901733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:48.334906 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.334863    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:50.334848619 +0000 UTC m=+5.993957365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:48.435878 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.435238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:48.435878 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.435433    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:48.435878 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.435453    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:48.435878 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.435466    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:48.435878 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.435525    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:17:50.435507921 +0000 UTC m=+6.094616677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:48.757555 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.757433    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:46 +0000 UTC" deadline="2028-02-11 11:35:28.336041445 +0000 UTC"
Jul 07 02:17:48.757555 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.757499    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14025h17m39.578546982s"
Jul 07 02:17:48.845584 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.845525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:48.845769 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.845654    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:17:48.846057 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.846037    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:48.846155 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.846136    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:48.846224 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.846211    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:48.846340 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:48.846297    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:48.861302 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.861255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9vrf7" event={"ID":"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6","Type":"ContainerStarted","Data":"4b0c049be5f53d76eeefe49430d66bebea40e98c526df51ae6d1f1f58fcd57ad"}
Jul 07 02:17:48.874364 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.874336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9mtv" event={"ID":"a1868bb6-d73c-4ac4-b26a-cd9e60b97515","Type":"ContainerStarted","Data":"4c0a98c23a64bbd8eb4f111286b619ebcf613c60459f98a7732868a6e572942f"}
Jul 07 02:17:48.896443 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.896414    2566 generic.go:356] "Generic (PLEG): container finished" podID="d0ca3d7c0f62b242a594ca6f1f035eef" containerID="bf72ae7ea318ddacd0d285515c48285466971e1e38c79b8fb906e58dc3b95047" exitCode=0
Jul 07 02:17:48.896649 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.896615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal" event={"ID":"d0ca3d7c0f62b242a594ca6f1f035eef","Type":"ContainerDied","Data":"bf72ae7ea318ddacd0d285515c48285466971e1e38c79b8fb906e58dc3b95047"}
Jul 07 02:17:48.905461 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.905078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal" event={"ID":"4b0b802c6511aef6dcda6b6d308a1cf9","Type":"ContainerStarted","Data":"f0a2d386f8758f388c5ef034f46f7bac7c21c7a9690bcb39291d085437b6deaa"}
Jul 07 02:17:48.929376 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.929349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lv8qk" event={"ID":"d0a8096f-0d4c-45c5-a548-316323d7dc8a","Type":"ContainerStarted","Data":"d1231cd63cb42a021ba00205ffa797b14c1b1b826c995f8525987540a31791ad"}
Jul 07 02:17:48.934902 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.934844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw" event={"ID":"cc8b27f6-e10e-4e0f-b760-9940e55c4416","Type":"ContainerStarted","Data":"d056c6339320fe332347b33524fd513caa391fc7dcbe89aa92af55b95d0314fa"}
Jul 07 02:17:48.944116 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.944088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phrvp" event={"ID":"af5d8a33-1406-4710-a9ed-3ce2b9e8788b","Type":"ContainerStarted","Data":"0b124d0fb5b47277d5be32e66ec40b7ec62d7667dfa9a67961c463efcb678486"}
Jul 07 02:17:48.960373 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.960322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerStarted","Data":"eca75dbaccac4713e0c731e505504414d22f7ea17c07924a1f80cb366c45db43"}
Jul 07 02:17:48.975038 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:48.974994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7gkcn" event={"ID":"9bc7941d-4a32-488e-9c02-cb43da0a5d04","Type":"ContainerStarted","Data":"5ab31b3e923997305455ed058e45661e2d96fcee92e68c2e71180edd48f3fe64"}
Jul 07 02:17:49.988467 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:49.988433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal" event={"ID":"d0ca3d7c0f62b242a594ca6f1f035eef","Type":"ContainerStarted","Data":"07ccd6671b9bf20e6772d44e26334a6cc44c3cefdb3abaac9514248838fb6886"}
Jul 07 02:17:50.002890 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.002819    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-238.ec2.internal" podStartSLOduration=5.002794052 podStartE2EDuration="5.002794052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:17:48.925098525 +0000 UTC m=+4.584207290" watchObservedRunningTime="2026-07-07 02:17:50.002794052 +0000 UTC m=+5.661902815"
Jul 07 02:17:50.357488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.357399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:50.357718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.357497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:50.357718 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.357633    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:50.357837 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.357774    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:50.358058 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.357979    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:54.357692459 +0000 UTC m=+10.016801215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:50.358058 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.358004    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:54.357993817 +0000 UTC m=+10.017102563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:50.458080 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.458015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:50.458333 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.458235    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:50.458333 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.458263    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:50.458333 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.458275    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:50.458333 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.458332    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:17:54.458313397 +0000 UTC m=+10.117422158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:50.842711 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.842613    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:50.842859 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.842746    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:50.842859 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.842811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:50.842960 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.842897    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:50.842960 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:50.842930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:50.844531 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:50.844498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:17:52.843981 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:52.843945    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:52.844459 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:52.844076    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:52.844459 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:52.844427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:52.844587 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:52.844533    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:52.844638 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:52.844616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:52.844744 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:52.844718    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:17:54.394040 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:54.394002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:54.394505 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:54.394056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:54.394505 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.394159    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:54.394505 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.394159    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:54.394505 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.394222    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:02.394203377 +0000 UTC m=+18.053312121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:54.394505 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.394241    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:02.39423187 +0000 UTC m=+18.053340627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:54.494863 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:54.494813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:54.495046 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.494999    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:54.495046 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.495021    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:54.495046 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.495036    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:54.495198 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.495091    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:18:02.495076094 +0000 UTC m=+18.154184834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:54.843605 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:54.842913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:54.843605 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.843023    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:17:54.843605 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:54.843365    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:54.843605 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.843466    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:54.843605 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:54.843508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:54.843605 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:54.843569    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:56.842562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:56.842515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:56.843045 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:56.842635    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:56.843045 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:56.842515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:56.843045 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:56.842705    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:56.843045 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:56.842802    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:17:56.843045 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:56.842893    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:58.842431 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:58.842400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:17:58.842969 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:58.842438    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:17:58.842969 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:17:58.842409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:17:58.842969 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:58.842511    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:17:58.842969 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:58.842574    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:17:58.842969 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:17:58.842648    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:00.843280 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:00.843244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:00.843716 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:00.843290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:00.843716 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:00.843254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:00.843716 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:00.843348    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:00.843716 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:00.843459    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:00.843716 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:00.843535    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:02.450425 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:02.450389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:02.450425 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:02.450435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:02.450986 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.450537    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:02.450986 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.450545    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:02.450986 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.450618    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.450597339 +0000 UTC m=+34.109706082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:02.450986 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.450709    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.450686881 +0000 UTC m=+34.109795623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:02.551049 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:02.551012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:02.551218 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.551132    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:02.551218 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.551151    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:02.551218 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.551164    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:02.551341 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.551219    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.551206318 +0000 UTC m=+34.210315058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:02.842406 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:02.842373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:02.842601 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:02.842476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:02.842601 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.842492    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:02.842601 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.842592    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:02.842799 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:02.842642    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:02.842799 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:02.842740    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:04.843025 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:04.842990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:04.843480 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:04.843077    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:04.843480 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:04.843091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:04.843480 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:04.843113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:04.843480 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:04.843222    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:04.843480 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:04.843309    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:06.843918 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:06.843372    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:06.843918 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:06.843903    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:06.845450 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:06.843575    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:06.845450 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:06.844005    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:06.845450 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:06.843528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:06.845450 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:06.844097    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:07.024622 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.024589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r9mtv" event={"ID":"a1868bb6-d73c-4ac4-b26a-cd9e60b97515","Type":"ContainerStarted","Data":"15f2f266fa7f2a72069ad1ac86ae6c5d5cb1d260a853d7c38202ba827fe56b2e"}
Jul 07 02:18:07.026120 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.026095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lv8qk" event={"ID":"d0a8096f-0d4c-45c5-a548-316323d7dc8a","Type":"ContainerStarted","Data":"58dc2a0057ea11fae72945c9fda36cad12dc6d9d11367c90616c9512418af5cb"}
Jul 07 02:18:07.027583 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.027561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw" event={"ID":"cc8b27f6-e10e-4e0f-b760-9940e55c4416","Type":"ContainerStarted","Data":"874df6cc40bb888cfeaeda4812fc89584a5d76122c6d2f05ae13ea2514b3083c"}
Jul 07 02:18:07.029124 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.029101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvhhh" event={"ID":"cc440849-966b-4c47-8e20-36454af21e2c","Type":"ContainerStarted","Data":"76b7868106d844653c7a1cb36ddc2d24dd1997c111b30cb0456c8af3d705595a"}
Jul 07 02:18:07.030500 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.030474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"f9f6e1e3e6c22eaf0c82e1acc51aa2885093cc0f68372a11f3cfbb0b4d4243b3"}
Jul 07 02:18:07.031885 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.031866    2566 generic.go:356] "Generic (PLEG): container finished" podID="c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e" containerID="3798c1b4d538603d444ddcbfc7cc75982995e5e90a380b1c5769d00807396c8f" exitCode=0
Jul 07 02:18:07.031981 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.031944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerDied","Data":"3798c1b4d538603d444ddcbfc7cc75982995e5e90a380b1c5769d00807396c8f"}
Jul 07 02:18:07.033411 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.033380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7gkcn" event={"ID":"9bc7941d-4a32-488e-9c02-cb43da0a5d04","Type":"ContainerStarted","Data":"b155114de82653ae1b8a54df561dc56b0961118242e3adbb1be738ca98215c17"}
Jul 07 02:18:07.035137 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.035113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9vrf7" event={"ID":"2054e60e-8de7-4424-a6fc-bccb5d0fe9c6","Type":"ContainerStarted","Data":"8c06c6d6ee797a84f690c85b5aba99ff139cafc8acd137db9ae99dd7f21ad342"}
Jul 07 02:18:07.040031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.039993    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r9mtv" podStartSLOduration=3.269784751 podStartE2EDuration="22.039982548s" totalImagesPullingTime="18.770197797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.039406878 +0000 UTC m=+22.698515637" watchObservedRunningTime="2026-07-07 02:18:07.039982548 +0000 UTC m=+22.699091310"
Jul 07 02:18:07.040606 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.040573    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-238.ec2.internal" podStartSLOduration=22.040562052 podStartE2EDuration="22.040562052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:17:50.003777978 +0000 UTC m=+5.662886743" watchObservedRunningTime="2026-07-07 02:18:07.040562052 +0000 UTC m=+22.699670817"
Jul 07 02:18:07.056141 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.056110    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9vrf7" podStartSLOduration=3.27057832 podStartE2EDuration="22.056098481s" totalImagesPullingTime="18.785520161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.055602761 +0000 UTC m=+22.714711523" watchObservedRunningTime="2026-07-07 02:18:07.056098481 +0000 UTC m=+22.715207242"
Jul 07 02:18:07.070623 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.070591    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jvhhh" podStartSLOduration=4.327774179 podStartE2EDuration="23.070579962s" totalImagesPullingTime="18.742805783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.06999993 +0000 UTC m=+22.729108692" watchObservedRunningTime="2026-07-07 02:18:07.070579962 +0000 UTC m=+22.729688724"
Jul 07 02:18:07.123204 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.123165    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7gkcn" podStartSLOduration=4.356631935 podStartE2EDuration="23.123153034s" totalImagesPullingTime="18.766521099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.123051005 +0000 UTC m=+22.782159768" watchObservedRunningTime="2026-07-07 02:18:07.123153034 +0000 UTC m=+22.782261796"
Jul 07 02:18:07.123566 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:07.123546    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lv8qk" podStartSLOduration=9.117966273 podStartE2EDuration="23.123540716s" totalImagesPullingTime="14.005574443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:07.105006704 +0000 UTC m=+22.764115466" watchObservedRunningTime="2026-07-07 02:18:07.123540716 +0000 UTC m=+22.782649477"
Jul 07 02:18:08.040717 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.040514    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:18:08.041329 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.041027    2566 generic.go:356] "Generic (PLEG): container finished" podID="327320b3-4c10-453f-811b-48f5033b92ce" containerID="06b95b67765540b6a63764c430260ab10b0bec0379c0fddb4f349f48b0556cef" exitCode=1
Jul 07 02:18:08.041329 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.041091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"11b27d3eab88c3641900cf9ac9ed403ba6d61faee0b7d863914272afed120745"}
Jul 07 02:18:08.041329 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.041192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"2f32b8168992caedcd6efa1836708c44bcd463bd8300f01e0400f29eea8db536"}
Jul 07 02:18:08.041329 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.041234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"6d60552f9367167995c388d43f196a60c12667b5737b71c6fe98ad240e84c7f1"}
Jul 07 02:18:08.041329 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.041266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"6a708e9f5afffc3fc6d4344f3fd12f952c145461e6c7d36088c62a6355fe5f5b"}
Jul 07 02:18:08.041329 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.041306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerDied","Data":"06b95b67765540b6a63764c430260ab10b0bec0379c0fddb4f349f48b0556cef"}
Jul 07 02:18:08.042596 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.042561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phrvp" event={"ID":"af5d8a33-1406-4710-a9ed-3ce2b9e8788b","Type":"ContainerStarted","Data":"a07217412f30e54e48a98318f63538b6cb41fc2ef8c318bb11bb23deb91c73c6"}
Jul 07 02:18:08.058056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.057999    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-phrvp" podStartSLOduration=4.295250672 podStartE2EDuration="23.057984576s" totalImagesPullingTime="18.762733904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.057906915 +0000 UTC m=+23.717015678" watchObservedRunningTime="2026-07-07 02:18:08.057984576 +0000 UTC m=+23.717093338"
Jul 07 02:18:08.395967 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.395934    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 02:18:08.792271 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.792179    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T02:18:08.395959559Z","UUID":"d9bebb90-4edf-4f8e-b9e2-4cf1238f0677","Handler":null,"Name":"","Endpoint":""}
Jul 07 02:18:08.793596 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.793578    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 02:18:08.793732 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.793610    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 02:18:08.842519 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.842486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:08.842519 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.842501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:08.842801 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:08.842499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:08.842801 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:08.842606    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:08.842801 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:08.842707    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:08.842949 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:08.842803    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:09.046607 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:09.046532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw" event={"ID":"cc8b27f6-e10e-4e0f-b760-9940e55c4416","Type":"ContainerStarted","Data":"76ea411d38617aab24c864d631af3e56949b45dffc718070ce24a83dd4bb8269"}
Jul 07 02:18:10.050291 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:10.050063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw" event={"ID":"cc8b27f6-e10e-4e0f-b760-9940e55c4416","Type":"ContainerStarted","Data":"871cd1e16580ac77a05b3ed261555d0bc730ec74ec5ade3dbfbdfc46a20307b5"}
Jul 07 02:18:10.053340 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:10.053322    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:18:10.053778 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:10.053753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"37d147633b89becb5ab344aa8e7161386da5d089c0b73598b1e88961e1f219aa"}
Jul 07 02:18:10.842873 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:10.842842    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:10.843080 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:10.842841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:10.843080 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:10.842980    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:10.843080 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:10.843044    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:10.843080 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:10.843060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:10.843266 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:10.843156    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:11.810345 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:11.810317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:18:11.811056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:11.811016    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:18:11.832088 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:11.832051    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4fw" podStartSLOduration=6.394053867 podStartE2EDuration="27.832037171s" totalImagesPullingTime="21.437983304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:10.067221304 +0000 UTC m=+25.726330067" watchObservedRunningTime="2026-07-07 02:18:11.832037171 +0000 UTC m=+27.491145937"
Jul 07 02:18:12.062448 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.062203    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:18:12.062806 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.062769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"630ea1939f001b08fdc135cf596be061ab666f760b26c7464c9d787875e49c67"}
Jul 07 02:18:12.063100 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.063049    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:18:12.063268 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.063250    2566 scope.go:117] "RemoveContainer" containerID="06b95b67765540b6a63764c430260ab10b0bec0379c0fddb4f349f48b0556cef"
Jul 07 02:18:12.063661 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.063641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lv8qk"
Jul 07 02:18:12.846745 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.846715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:12.846745 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.846731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:12.847303 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:12.846816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:12.847303 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:12.846828    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:12.847303 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:12.846932    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:12.847303 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:12.847005    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:14.071160 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.071003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:18:14.071532 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.071509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" event={"ID":"327320b3-4c10-453f-811b-48f5033b92ce","Type":"ContainerStarted","Data":"73fd69b575ee7e85c2827229434f1aab6cde5c36faa1227052aed91a2b9202ea"}
Jul 07 02:18:14.071631 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.071616    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:18:14.071777 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.071758    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:18:14.071928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.071819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:18:14.073524 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.073500    2566 generic.go:356] "Generic (PLEG): container finished" podID="c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e" containerID="f76c324f37f2e1baaf4bfe1ace56f402faab9a4da27d246de674ca03169a888a" exitCode=0
Jul 07 02:18:14.073639 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.073575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerDied","Data":"f76c324f37f2e1baaf4bfe1ace56f402faab9a4da27d246de674ca03169a888a"}
Jul 07 02:18:14.088564 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.088502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:18:14.088708 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.088599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:18:14.104590 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.104544    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" podStartSLOduration=10.140772866 podStartE2EDuration="29.104529285s" totalImagesPullingTime="18.963756419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:14.103050507 +0000 UTC m=+29.762159268" watchObservedRunningTime="2026-07-07 02:18:14.104529285 +0000 UTC m=+29.763638046"
Jul 07 02:18:14.105735 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.105663    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2khl"]
Jul 07 02:18:14.105835 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.105800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:14.105933 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:14.105911    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:14.106430 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.106409    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ns6nx"]
Jul 07 02:18:14.106530 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.106484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:14.106595 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:14.106578    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:14.117720 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.117695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gvjbd"]
Jul 07 02:18:14.117815 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:14.117765    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:14.117853 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:14.117831    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:15.842559 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:15.842527    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:15.843060 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:15.842572    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:15.843060 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:15.842608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:15.843060 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:15.842627    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:15.843060 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:15.842731    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:15.843060 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:15.842807    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:16.080427 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:16.080339    2566 generic.go:356] "Generic (PLEG): container finished" podID="c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e" containerID="9b64ad1551e61ff435032f596ed47c2bd8f62d666c506f2d85cf07b96bb2b2a1" exitCode=0
Jul 07 02:18:16.080427 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:16.080381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerDied","Data":"9b64ad1551e61ff435032f596ed47c2bd8f62d666c506f2d85cf07b96bb2b2a1"}
Jul 07 02:18:16.090444 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:16.090412    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr" podUID="327320b3-4c10-453f-811b-48f5033b92ce" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 07 02:18:17.084605 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:17.084568    2566 generic.go:356] "Generic (PLEG): container finished" podID="c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e" containerID="ae03d9f443b6bd27a02675bf152ddec5213180c0e445d5690ea2ecb5da579b92" exitCode=0
Jul 07 02:18:17.085209 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:17.084620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerDied","Data":"ae03d9f443b6bd27a02675bf152ddec5213180c0e445d5690ea2ecb5da579b92"}
Jul 07 02:18:17.842870 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:17.842832    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:17.842870 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:17.842869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:17.843094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:17.842927    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:17.843094 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:17.842940    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:17.843094 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:17.843006    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:17.843094 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:17.843047    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:18.462150 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:18.462113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:18.462598 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:18.462166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:18.462598 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.462292    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:18.462598 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.462295    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:18.462598 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.462367    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.462345762 +0000 UTC m=+66.121454503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:18.462598 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.462387    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret podName:c3cee6f8-980e-4530-b615-b02786a8e117 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.462377549 +0000 UTC m=+66.121486295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret") pod "global-pull-secret-syncer-gvjbd" (UID: "c3cee6f8-980e-4530-b615-b02786a8e117") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:18.563084 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:18.563049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:18.563272 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.563250    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:18.563335 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.563281    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:18.563335 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.563297    2566 projected.go:196] Error preparing data for projected volume kube-api-access-d5727 for pod openshift-network-diagnostics/network-check-target-ns6nx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:18.563430 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:18.563362    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727 podName:bd3f6020-1ea4-4a96-bc08-7d370557b27d nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.563342788 +0000 UTC m=+66.222451529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5727" (UniqueName: "kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727") pod "network-check-target-ns6nx" (UID: "bd3f6020-1ea4-4a96-bc08-7d370557b27d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:19.842608 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:19.842547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:19.843123 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:19.842573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:19.843123 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:19.842701    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvjbd" podUID="c3cee6f8-980e-4530-b615-b02786a8e117"
Jul 07 02:18:19.843123 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:19.842728    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:19.843123 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:19.842824    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ns6nx" podUID="bd3f6020-1ea4-4a96-bc08-7d370557b27d"
Jul 07 02:18:19.843123 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:19.842900    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2khl" podUID="eb530956-7b82-4d88-975f-3c3e042b7fb8"
Jul 07 02:18:20.605068 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.605036    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-238.ec2.internal" event="NodeReady"
Jul 07 02:18:20.605336 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.605190    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 02:18:20.653861 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.653827    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wttkz"]
Jul 07 02:18:20.657953 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.657925    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-br5cd"]
Jul 07 02:18:20.658134 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.658051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.660710 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.660687    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 02:18:20.660877 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.660856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.661049 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.661028    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc8t9\""
Jul 07 02:18:20.661538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.661515    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 02:18:20.663032 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.663014    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 02:18:20.663422 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.663405    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 02:18:20.663503 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.663483    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 02:18:20.663878 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.663859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mfm2n\""
Jul 07 02:18:20.667776 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.667499    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wttkz"]
Jul 07 02:18:20.681436 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.681414    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-br5cd"]
Jul 07 02:18:20.780646 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.780610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43f4893-fdbb-422f-af88-799cb1e33ec6-tmp-dir\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.780646 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.780653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rv5\" (UniqueName: \"kubernetes.io/projected/c43f4893-fdbb-422f-af88-799cb1e33ec6-kube-api-access-m6rv5\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.780890 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.780717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61893ef3-a907-4929-9848-e0fee89fd5d9-cert\") pod \"ingress-canary-br5cd\" (UID: \"61893ef3-a907-4929-9848-e0fee89fd5d9\") " pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.780890 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.780816    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c43f4893-fdbb-422f-af88-799cb1e33ec6-config-volume\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.780890 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.780854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.781026 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.780893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcm2\" (UniqueName: \"kubernetes.io/projected/61893ef3-a907-4929-9848-e0fee89fd5d9-kube-api-access-cpcm2\") pod \"ingress-canary-br5cd\" (UID: \"61893ef3-a907-4929-9848-e0fee89fd5d9\") " pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.882008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.881922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61893ef3-a907-4929-9848-e0fee89fd5d9-cert\") pod \"ingress-canary-br5cd\" (UID: \"61893ef3-a907-4929-9848-e0fee89fd5d9\") " pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.882008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.881986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c43f4893-fdbb-422f-af88-799cb1e33ec6-config-volume\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.882533 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.882010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.882533 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.882042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcm2\" (UniqueName: \"kubernetes.io/projected/61893ef3-a907-4929-9848-e0fee89fd5d9-kube-api-access-cpcm2\") pod \"ingress-canary-br5cd\" (UID: \"61893ef3-a907-4929-9848-e0fee89fd5d9\") " pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.882533 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.882088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43f4893-fdbb-422f-af88-799cb1e33ec6-tmp-dir\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.882533 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.882117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rv5\" (UniqueName: \"kubernetes.io/projected/c43f4893-fdbb-422f-af88-799cb1e33ec6-kube-api-access-m6rv5\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.882533 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:20.882180    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:20.882533 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:20.882251    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls podName:c43f4893-fdbb-422f-af88-799cb1e33ec6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:21.382229553 +0000 UTC m=+37.041338307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls") pod "dns-default-wttkz" (UID: "c43f4893-fdbb-422f-af88-799cb1e33ec6") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:20.882826 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.882570    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43f4893-fdbb-422f-af88-799cb1e33ec6-tmp-dir\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.882884 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.882713    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c43f4893-fdbb-422f-af88-799cb1e33ec6-config-volume\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.887164 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.887139    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61893ef3-a907-4929-9848-e0fee89fd5d9-cert\") pod \"ingress-canary-br5cd\" (UID: \"61893ef3-a907-4929-9848-e0fee89fd5d9\") " pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.891052 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.890997    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rv5\" (UniqueName: \"kubernetes.io/projected/c43f4893-fdbb-422f-af88-799cb1e33ec6-kube-api-access-m6rv5\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:20.891194 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.891096    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcm2\" (UniqueName: \"kubernetes.io/projected/61893ef3-a907-4929-9848-e0fee89fd5d9-kube-api-access-cpcm2\") pod \"ingress-canary-br5cd\" (UID: \"61893ef3-a907-4929-9848-e0fee89fd5d9\") " pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:20.977130 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:20.977095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-br5cd"
Jul 07 02:18:21.386321 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.386286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:21.386511 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:21.386450    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:21.386580 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:21.386517    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls podName:c43f4893-fdbb-422f-af88-799cb1e33ec6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:22.386500398 +0000 UTC m=+38.045609137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls") pod "dns-default-wttkz" (UID: "c43f4893-fdbb-422f-af88-799cb1e33ec6") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:21.843278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.843229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:21.843278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.843259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:21.843523 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.843259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:21.846328 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.846290    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:18:21.846328 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.846284    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:18:21.846328 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.846337    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ggshj\""
Jul 07 02:18:21.846586 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.846416    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hf7w2\""
Jul 07 02:18:21.846586 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.846343    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:18:21.846586 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:21.846586    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:18:22.231915 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:22.231884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-br5cd"]
Jul 07 02:18:22.263274 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:22.263242    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61893ef3_a907_4929_9848_e0fee89fd5d9.slice/crio-2d39ae82c7dc503f29dee844865e6816991b18cca58146bb582ae93e11a93d64 WatchSource:0}: Error finding container 2d39ae82c7dc503f29dee844865e6816991b18cca58146bb582ae93e11a93d64: Status 404 returned error can't find the container with id 2d39ae82c7dc503f29dee844865e6816991b18cca58146bb582ae93e11a93d64
Jul 07 02:18:22.396351 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:22.396228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:22.396351 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:22.396335    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:22.396485 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:22.396390    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls podName:c43f4893-fdbb-422f-af88-799cb1e33ec6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:24.396375843 +0000 UTC m=+40.055484583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls") pod "dns-default-wttkz" (UID: "c43f4893-fdbb-422f-af88-799cb1e33ec6") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:23.099494 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:23.099449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-br5cd" event={"ID":"61893ef3-a907-4929-9848-e0fee89fd5d9","Type":"ContainerStarted","Data":"2d39ae82c7dc503f29dee844865e6816991b18cca58146bb582ae93e11a93d64"}
Jul 07 02:18:23.102597 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:23.102565    2566 generic.go:356] "Generic (PLEG): container finished" podID="c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e" containerID="f61a43a8b7f47f667764d14d9c2449fd2552bd0b8e42bb0d9ba88edacd51ffa2" exitCode=0
Jul 07 02:18:23.102743 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:23.102615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerDied","Data":"f61a43a8b7f47f667764d14d9c2449fd2552bd0b8e42bb0d9ba88edacd51ffa2"}
Jul 07 02:18:24.108921 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:24.108685    2566 generic.go:356] "Generic (PLEG): container finished" podID="c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e" containerID="6d305781204ead3516f97a20a59505ce908410ebbfe263bbfea67b3c1b2905f7" exitCode=0
Jul 07 02:18:24.108921 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:24.108712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerDied","Data":"6d305781204ead3516f97a20a59505ce908410ebbfe263bbfea67b3c1b2905f7"}
Jul 07 02:18:24.408611 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:24.408531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:24.408762 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:24.408623    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:24.408762 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:24.408700    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls podName:c43f4893-fdbb-422f-af88-799cb1e33ec6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:28.408666294 +0000 UTC m=+44.067775033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls") pod "dns-default-wttkz" (UID: "c43f4893-fdbb-422f-af88-799cb1e33ec6") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:25.112562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:25.112473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-br5cd" event={"ID":"61893ef3-a907-4929-9848-e0fee89fd5d9","Type":"ContainerStarted","Data":"6ef8ab54675a93295d21da27a55ab4e500d1d78848a7147a0bb9c5395aa96bac"}
Jul 07 02:18:25.115678 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:25.115648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" event={"ID":"c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e","Type":"ContainerStarted","Data":"88da891a0138264a4c1c7d2892e533b2e7a301d23d1ad9165f5972592fe2fb5e"}
Jul 07 02:18:25.129447 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:25.129392    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-br5cd" podStartSLOduration=2.626135631 podStartE2EDuration="5.12937528s" totalImagesPullingTime="2.503239649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:25.128712827 +0000 UTC m=+40.787821591" watchObservedRunningTime="2026-07-07 02:18:25.12937528 +0000 UTC m=+40.788484044"
Jul 07 02:18:25.154016 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:25.153967    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r2x6b" podStartSLOduration=8.925615125 podStartE2EDuration="41.153953637s" totalImagesPullingTime="32.228338512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:25.15292565 +0000 UTC m=+40.812034403" watchObservedRunningTime="2026-07-07 02:18:25.153953637 +0000 UTC m=+40.813062398"
Jul 07 02:18:28.439290 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:28.439248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:28.439715 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:28.439401    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:28.439715 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:28.439473    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls podName:c43f4893-fdbb-422f-af88-799cb1e33ec6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:36.439454501 +0000 UTC m=+52.098563256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls") pod "dns-default-wttkz" (UID: "c43f4893-fdbb-422f-af88-799cb1e33ec6") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:36.494945 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:36.494904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:36.495399 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:36.495045    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:36.495399 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:36.495107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls podName:c43f4893-fdbb-422f-af88-799cb1e33ec6 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:52.495091828 +0000 UTC m=+68.154200568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls") pod "dns-default-wttkz" (UID: "c43f4893-fdbb-422f-af88-799cb1e33ec6") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:39.617158 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.617119    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"]
Jul 07 02:18:39.619214 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.619191    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-4mqhh"]
Jul 07 02:18:39.619346 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.619315    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.621055 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.621035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.624520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624430    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 07 02:18:39.624520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 02:18:39.624520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624484    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vs224\""
Jul 07 02:18:39.624520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 07 02:18:39.625004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624543    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 02:18:39.625004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 02:18:39.625004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624822    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 07 02:18:39.625004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624837    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 02:18:39.625004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.624842    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 07 02:18:39.625004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.625004    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6qhd9\""
Jul 07 02:18:39.629650 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.629632    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 07 02:18:39.636542 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.636518    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"]
Jul 07 02:18:39.637228 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.637208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-4mqhh"]
Jul 07 02:18:39.712570 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.712538    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"]
Jul 07 02:18:39.713858 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.713838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jfs\" (UniqueName: \"kubernetes.io/projected/4753d121-c72d-45fa-a249-ef8ccc73d525-kube-api-access-92jfs\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.713938 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.713886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4753d121-c72d-45fa-a249-ef8ccc73d525-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.713938 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.713909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.714016 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.713943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.714016 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.713993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-serving-cert\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.714016 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.714010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-tmp\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.714126 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.714028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-snapshots\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.714126 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.714083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.714126 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.714109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzf6\" (UniqueName: \"kubernetes.io/projected/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-kube-api-access-gfzf6\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.714285 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.714270    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.717002 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.716979    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 07 02:18:39.717121 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.717028    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 07 02:18:39.717301 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.717283    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t6m2g\""
Jul 07 02:18:39.717445 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.717425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 07 02:18:39.717510 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.717468    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:39.718529 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.718513    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-bktlx"]
Jul 07 02:18:39.720122 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.720106    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.722690 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.722493    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-clr6j\""
Jul 07 02:18:39.722690 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.722635    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 07 02:18:39.723488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.723158    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:39.723488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.723367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 07 02:18:39.724759 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.724744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 07 02:18:39.727218 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.727198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"]
Jul 07 02:18:39.729562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.729416    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 07 02:18:39.739365 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.739343    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-bktlx"]
Jul 07 02:18:39.815310 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-snapshots\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.815310 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d4475f0-c6bb-4743-9419-a2bb4df52d56-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.815534 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.815534 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-serving-cert\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.815534 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzf6\" (UniqueName: \"kubernetes.io/projected/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-kube-api-access-gfzf6\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.815534 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d4475f0-c6bb-4743-9419-a2bb4df52d56-config\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.815534 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcb6\" (UniqueName: \"kubernetes.io/projected/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-kube-api-access-bzcb6\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.815809 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jfs\" (UniqueName: \"kubernetes.io/projected/4753d121-c72d-45fa-a249-ef8ccc73d525-kube-api-access-92jfs\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.815809 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4753d121-c72d-45fa-a249-ef8ccc73d525-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.815809 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.815959 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.815959 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-serving-cert\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.815959 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkng\" (UniqueName: \"kubernetes.io/projected/0d4475f0-c6bb-4743-9419-a2bb4df52d56-kube-api-access-hkkng\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.816108 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-config\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.816108 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.815993    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-snapshots\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.816108 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.816006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-trusted-ca\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.816108 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:39.815987    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:39.816308 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.816089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-tmp\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.816308 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:39.816169    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls podName:4753d121-c72d-45fa-a249-ef8ccc73d525 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:40.316147155 +0000 UTC m=+55.975255905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-5pqn5" (UID: "4753d121-c72d-45fa-a249-ef8ccc73d525") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:39.816429 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.816307    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.816429 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.816308    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-tmp\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.816429 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.816355    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.816556 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.816475    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4753d121-c72d-45fa-a249-ef8ccc73d525-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.819312 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.819296    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-serving-cert\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.831886 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.831864    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzf6\" (UniqueName: \"kubernetes.io/projected/f975e7d9-c1a7-41bd-89fb-6fe6f706aff6-kube-api-access-gfzf6\") pod \"insights-operator-5c847dcfc4-4mqhh\" (UID: \"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6\") " pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:39.832006 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.831867    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jfs\" (UniqueName: \"kubernetes.io/projected/4753d121-c72d-45fa-a249-ef8ccc73d525-kube-api-access-92jfs\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:39.916601 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d4475f0-c6bb-4743-9419-a2bb4df52d56-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.916601 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-serving-cert\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.916779 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d4475f0-c6bb-4743-9419-a2bb4df52d56-config\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.916779 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcb6\" (UniqueName: \"kubernetes.io/projected/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-kube-api-access-bzcb6\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.916891 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkng\" (UniqueName: \"kubernetes.io/projected/0d4475f0-c6bb-4743-9419-a2bb4df52d56-kube-api-access-hkkng\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.916891 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-config\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.916993 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.916903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-trusted-ca\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.917300 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.917270    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d4475f0-c6bb-4743-9419-a2bb4df52d56-config\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.917515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.917496    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-config\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.917715 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.917699    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-trusted-ca\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.918537 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.918516    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d4475f0-c6bb-4743-9419-a2bb4df52d56-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.918773 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.918753    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-serving-cert\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.926815 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.926795    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkng\" (UniqueName: \"kubernetes.io/projected/0d4475f0-c6bb-4743-9419-a2bb4df52d56-kube-api-access-hkkng\") pod \"service-ca-operator-5d46d9ffd4-78fnh\" (UID: \"0d4475f0-c6bb-4743-9419-a2bb4df52d56\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:39.927010 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.926994    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcb6\" (UniqueName: \"kubernetes.io/projected/4b6fc827-c7ca-4b38-8aef-63d8b12166bd-kube-api-access-bzcb6\") pod \"console-operator-68b757fb67-bktlx\" (UID: \"4b6fc827-c7ca-4b38-8aef-63d8b12166bd\") " pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:39.934926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:39.934909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh"
Jul 07 02:18:40.024229 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.024176    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"
Jul 07 02:18:40.032248 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.032223    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:40.056622 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.056588    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-4mqhh"]
Jul 07 02:18:40.060283 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:40.060248    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf975e7d9_c1a7_41bd_89fb_6fe6f706aff6.slice/crio-2e1d91b95755e93b5bc5e4ae46c204f506169c9ccbd74681e1380c9bc38b28db WatchSource:0}: Error finding container 2e1d91b95755e93b5bc5e4ae46c204f506169c9ccbd74681e1380c9bc38b28db: Status 404 returned error can't find the container with id 2e1d91b95755e93b5bc5e4ae46c204f506169c9ccbd74681e1380c9bc38b28db
Jul 07 02:18:40.152148 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.152117    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh"]
Jul 07 02:18:40.152539 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.152509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh" event={"ID":"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6","Type":"ContainerStarted","Data":"2e1d91b95755e93b5bc5e4ae46c204f506169c9ccbd74681e1380c9bc38b28db"}
Jul 07 02:18:40.154969 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:40.154949    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4475f0_c6bb_4743_9419_a2bb4df52d56.slice/crio-c53ed347ad46d9d22220a7a382fe7656bc60e107dc623c70ddda1ed3d53fdcae WatchSource:0}: Error finding container c53ed347ad46d9d22220a7a382fe7656bc60e107dc623c70ddda1ed3d53fdcae: Status 404 returned error can't find the container with id c53ed347ad46d9d22220a7a382fe7656bc60e107dc623c70ddda1ed3d53fdcae
Jul 07 02:18:40.168548 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.168493    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-bktlx"]
Jul 07 02:18:40.171149 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:40.171125    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6fc827_c7ca_4b38_8aef_63d8b12166bd.slice/crio-75d16c4ee50a13c3c3b52c9874526127775a4d51ed0693d33f866ee883d08ee2 WatchSource:0}: Error finding container 75d16c4ee50a13c3c3b52c9874526127775a4d51ed0693d33f866ee883d08ee2: Status 404 returned error can't find the container with id 75d16c4ee50a13c3c3b52c9874526127775a4d51ed0693d33f866ee883d08ee2
Jul 07 02:18:40.319913 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:40.319878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:40.320056 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:40.319982    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:40.320056 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:40.320038    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls podName:4753d121-c72d-45fa-a249-ef8ccc73d525 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:41.320024249 +0000 UTC m=+56.979132989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-5pqn5" (UID: "4753d121-c72d-45fa-a249-ef8ccc73d525") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:41.157326 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:41.157250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" event={"ID":"4b6fc827-c7ca-4b38-8aef-63d8b12166bd","Type":"ContainerStarted","Data":"75d16c4ee50a13c3c3b52c9874526127775a4d51ed0693d33f866ee883d08ee2"}
Jul 07 02:18:41.159078 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:41.159016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh" event={"ID":"0d4475f0-c6bb-4743-9419-a2bb4df52d56","Type":"ContainerStarted","Data":"c53ed347ad46d9d22220a7a382fe7656bc60e107dc623c70ddda1ed3d53fdcae"}
Jul 07 02:18:41.327986 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:41.327947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:41.328145 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:41.328068    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:41.328145 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:41.328121    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls podName:4753d121-c72d-45fa-a249-ef8ccc73d525 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:43.328107106 +0000 UTC m=+58.987215847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-5pqn5" (UID: "4753d121-c72d-45fa-a249-ef8ccc73d525") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:43.344002 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.343973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:43.344340 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:43.344121    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:43.344340 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:43.344182    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls podName:4753d121-c72d-45fa-a249-ef8ccc73d525 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:47.344164034 +0000 UTC m=+63.003272779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-5pqn5" (UID: "4753d121-c72d-45fa-a249-ef8ccc73d525") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:43.759088 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.759050    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"]
Jul 07 02:18:43.761061 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.761044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"
Jul 07 02:18:43.763658 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.763629    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:43.763993 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.763974    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5ndmk\""
Jul 07 02:18:43.764632 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.764617    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 02:18:43.772863 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.772837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"]
Jul 07 02:18:43.849100 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.849039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgdl\" (UniqueName: \"kubernetes.io/projected/c7544dc0-bff1-42bd-b00d-a8911eee438f-kube-api-access-xbgdl\") pod \"migrator-d57d58cc9-n9t4p\" (UID: \"c7544dc0-bff1-42bd-b00d-a8911eee438f\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"
Jul 07 02:18:43.950311 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.950281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgdl\" (UniqueName: \"kubernetes.io/projected/c7544dc0-bff1-42bd-b00d-a8911eee438f-kube-api-access-xbgdl\") pod \"migrator-d57d58cc9-n9t4p\" (UID: \"c7544dc0-bff1-42bd-b00d-a8911eee438f\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"
Jul 07 02:18:43.958364 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:43.958332    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgdl\" (UniqueName: \"kubernetes.io/projected/c7544dc0-bff1-42bd-b00d-a8911eee438f-kube-api-access-xbgdl\") pod \"migrator-d57d58cc9-n9t4p\" (UID: \"c7544dc0-bff1-42bd-b00d-a8911eee438f\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"
Jul 07 02:18:44.066636 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.066556    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-swmdm"]
Jul 07 02:18:44.068811 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.068779    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.069737 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.069719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"
Jul 07 02:18:44.072221 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.072199    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 02:18:44.072930 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.072907    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vqqxg\""
Jul 07 02:18:44.073138 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.073115    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 02:18:44.091295 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.091256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-swmdm"]
Jul 07 02:18:44.152025 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.151947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-crio-socket\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.152025 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.151990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-data-volume\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.152259 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.152069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.152259 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.152099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcbz\" (UniqueName: \"kubernetes.io/projected/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-api-access-spcbz\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.152259 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.152125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.169168 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.169076    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/0.log"
Jul 07 02:18:44.169168 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.169120    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b6fc827-c7ca-4b38-8aef-63d8b12166bd" containerID="628ef83415f974ed9656b201b9ea9ea9273e438700916b205cde7149fba37aa6" exitCode=255
Jul 07 02:18:44.169343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.169220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" event={"ID":"4b6fc827-c7ca-4b38-8aef-63d8b12166bd","Type":"ContainerDied","Data":"628ef83415f974ed9656b201b9ea9ea9273e438700916b205cde7149fba37aa6"}
Jul 07 02:18:44.169615 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.169598    2566 scope.go:117] "RemoveContainer" containerID="628ef83415f974ed9656b201b9ea9ea9273e438700916b205cde7149fba37aa6"
Jul 07 02:18:44.171474 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.171439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh" event={"ID":"0d4475f0-c6bb-4743-9419-a2bb4df52d56","Type":"ContainerStarted","Data":"960ac7c676022830cd08237ab7bb1e069682f7acd1c1ec28ddd12629d21e16d0"}
Jul 07 02:18:44.178773 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.178739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh" event={"ID":"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6","Type":"ContainerStarted","Data":"17f7f8d85d23341ac54749de6848345f5fdf0d455260fac219d7a643d7d15d4c"}
Jul 07 02:18:44.203980 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.203931    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh" podStartSLOduration=2.115487417 podStartE2EDuration="5.203910093s" totalImagesPullingTime="3.088422676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:44.203749592 +0000 UTC m=+59.862858354" watchObservedRunningTime="2026-07-07 02:18:44.203910093 +0000 UTC m=+59.863018857"
Jul 07 02:18:44.208427 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.208369    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p"]
Jul 07 02:18:44.213867 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:44.212913    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7544dc0_bff1_42bd_b00d_a8911eee438f.slice/crio-46004e1df709f54a17c0f3be514c9da75b9e9553520979cb3321a63b8831c6d8 WatchSource:0}: Error finding container 46004e1df709f54a17c0f3be514c9da75b9e9553520979cb3321a63b8831c6d8: Status 404 returned error can't find the container with id 46004e1df709f54a17c0f3be514c9da75b9e9553520979cb3321a63b8831c6d8
Jul 07 02:18:44.220930 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.220879    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh" podStartSLOduration=2.232135476 podStartE2EDuration="5.220860532s" totalImagesPullingTime="2.988725056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:44.220438797 +0000 UTC m=+59.879547559" watchObservedRunningTime="2026-07-07 02:18:44.220860532 +0000 UTC m=+59.879969293"
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.252545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.252605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcbz\" (UniqueName: \"kubernetes.io/projected/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-api-access-spcbz\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.252634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.252824    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.252884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:44.752866468 +0000 UTC m=+60.411975229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.252959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-crio-socket\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.253017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-data-volume\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.253048    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.253293    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:44.753275012 +0000 UTC m=+60.412383771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : configmap "kube-rbac-proxy" not found
Jul 07 02:18:44.253407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.253352    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-crio-socket\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.254019 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.253466    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-data-volume\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.262019 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.261989    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcbz\" (UniqueName: \"kubernetes.io/projected/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-api-access-spcbz\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.757141 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.757095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.757141 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:44.757143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:44.757565 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.757233    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:18:44.757565 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.757245    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:44.757565 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.757291    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:45.757275644 +0000 UTC m=+61.416384386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : configmap "kube-rbac-proxy" not found
Jul 07 02:18:44.757565 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:44.757305    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:45.757299132 +0000 UTC m=+61.416407872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:45.183250 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.183180    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:18:45.183633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.183611    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/0.log"
Jul 07 02:18:45.183782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.183646    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b6fc827-c7ca-4b38-8aef-63d8b12166bd" containerID="0226e81644e05dadc8491a46aa3fe6b656e9ac6e139ca329687763ddcaf1fa4e" exitCode=255
Jul 07 02:18:45.183782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.183735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" event={"ID":"4b6fc827-c7ca-4b38-8aef-63d8b12166bd","Type":"ContainerDied","Data":"0226e81644e05dadc8491a46aa3fe6b656e9ac6e139ca329687763ddcaf1fa4e"}
Jul 07 02:18:45.183782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.183779    2566 scope.go:117] "RemoveContainer" containerID="628ef83415f974ed9656b201b9ea9ea9273e438700916b205cde7149fba37aa6"
Jul 07 02:18:45.183945 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.183930    2566 scope.go:117] "RemoveContainer" containerID="0226e81644e05dadc8491a46aa3fe6b656e9ac6e139ca329687763ddcaf1fa4e"
Jul 07 02:18:45.184117 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:45.184082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-bktlx_openshift-console-operator(4b6fc827-c7ca-4b38-8aef-63d8b12166bd)\"" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" podUID="4b6fc827-c7ca-4b38-8aef-63d8b12166bd"
Jul 07 02:18:45.185150 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.184818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p" event={"ID":"c7544dc0-bff1-42bd-b00d-a8911eee438f","Type":"ContainerStarted","Data":"46004e1df709f54a17c0f3be514c9da75b9e9553520979cb3321a63b8831c6d8"}
Jul 07 02:18:45.764551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.764518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:45.765004 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:45.764632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:45.765004 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:45.764656    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:18:45.765004 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:45.764733    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:45.765004 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:45.764765    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:47.76474232 +0000 UTC m=+63.423851060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : configmap "kube-rbac-proxy" not found
Jul 07 02:18:45.765004 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:45.764786    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:47.764776683 +0000 UTC m=+63.423885423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:46.090541 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.090515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g4dxr"
Jul 07 02:18:46.189472 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.189446    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:18:46.189836 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.189809    2566 scope.go:117] "RemoveContainer" containerID="0226e81644e05dadc8491a46aa3fe6b656e9ac6e139ca329687763ddcaf1fa4e"
Jul 07 02:18:46.190022 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:46.189997    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-bktlx_openshift-console-operator(4b6fc827-c7ca-4b38-8aef-63d8b12166bd)\"" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" podUID="4b6fc827-c7ca-4b38-8aef-63d8b12166bd"
Jul 07 02:18:46.191479 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.191457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p" event={"ID":"c7544dc0-bff1-42bd-b00d-a8911eee438f","Type":"ContainerStarted","Data":"dcc3235573630e967435b3c876bf84d3f51b2f8b0c3df8c3ae1469349444a439"}
Jul 07 02:18:46.191594 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.191498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p" event={"ID":"c7544dc0-bff1-42bd-b00d-a8911eee438f","Type":"ContainerStarted","Data":"0189dc993e81937847315688c923e877709dfeb2f1590d3ab84045be85d8df8f"}
Jul 07 02:18:46.224854 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.224792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-n9t4p" podStartSLOduration=1.487843241 podStartE2EDuration="3.224776065s" totalImagesPullingTime="1.736932824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:46.224037965 +0000 UTC m=+61.883146728" watchObservedRunningTime="2026-07-07 02:18:46.224776065 +0000 UTC m=+61.883884827"
Jul 07 02:18:46.955165 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.955128    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x9f6g"]
Jul 07 02:18:46.957368 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.957343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:46.959791 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.959767    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 02:18:46.959890 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.959830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 02:18:46.960755 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.960736    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 02:18:46.960863 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.960757    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lz4c7\""
Jul 07 02:18:46.960959 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.960896    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 02:18:46.967446 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.967429    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x9f6g"]
Jul 07 02:18:46.975519 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.975498    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ab3d260-6260-4cd4-9400-43a9bef9179b-signing-cabundle\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:46.975630 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.975527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ab3d260-6260-4cd4-9400-43a9bef9179b-signing-key\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:46.975630 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:46.975564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfl4h\" (UniqueName: \"kubernetes.io/projected/3ab3d260-6260-4cd4-9400-43a9bef9179b-kube-api-access-lfl4h\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.047904 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.047881    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9mtv_a1868bb6-d73c-4ac4-b26a-cd9e60b97515/dns-node-resolver/0.log"
Jul 07 02:18:47.076554 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.076529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ab3d260-6260-4cd4-9400-43a9bef9179b-signing-cabundle\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.076662 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.076558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ab3d260-6260-4cd4-9400-43a9bef9179b-signing-key\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.076662 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.076592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfl4h\" (UniqueName: \"kubernetes.io/projected/3ab3d260-6260-4cd4-9400-43a9bef9179b-kube-api-access-lfl4h\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.077297 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.077278    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3ab3d260-6260-4cd4-9400-43a9bef9179b-signing-cabundle\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.078939 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.078921    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3ab3d260-6260-4cd4-9400-43a9bef9179b-signing-key\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.085029 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.085006    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfl4h\" (UniqueName: \"kubernetes.io/projected/3ab3d260-6260-4cd4-9400-43a9bef9179b-kube-api-access-lfl4h\") pod \"service-ca-758f599c5-x9f6g\" (UID: \"3ab3d260-6260-4cd4-9400-43a9bef9179b\") " pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.266724 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.266666    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-x9f6g"
Jul 07 02:18:47.379774 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.379739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:47.379950 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:47.379871    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:47.379950 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:47.379942    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls podName:4753d121-c72d-45fa-a249-ef8ccc73d525 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:55.379922951 +0000 UTC m=+71.039031692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-5pqn5" (UID: "4753d121-c72d-45fa-a249-ef8ccc73d525") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:47.392355 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.392321    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-x9f6g"]
Jul 07 02:18:47.395500 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:47.395470    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab3d260_6260_4cd4_9400_43a9bef9179b.slice/crio-f8cf486c51330e78469c7ba6160ec4e2a073a3befa2fb84dce4b2efd0212343a WatchSource:0}: Error finding container f8cf486c51330e78469c7ba6160ec4e2a073a3befa2fb84dce4b2efd0212343a: Status 404 returned error can't find the container with id f8cf486c51330e78469c7ba6160ec4e2a073a3befa2fb84dce4b2efd0212343a
Jul 07 02:18:47.783877 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.783834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:47.783877 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:47.783880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:47.784098 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:47.783977    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:18:47.784098 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:47.784012    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:47.784098 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:47.784023    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:51.784010686 +0000 UTC m=+67.443119430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : configmap "kube-rbac-proxy" not found
Jul 07 02:18:47.784098 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:47.784085    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:51.784066705 +0000 UTC m=+67.443175452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:48.046600 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:48.046523    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvhhh_cc440849-966b-4c47-8e20-36454af21e2c/node-ca/0.log"
Jul 07 02:18:48.199409 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:48.199377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-x9f6g" event={"ID":"3ab3d260-6260-4cd4-9400-43a9bef9179b","Type":"ContainerStarted","Data":"daf2eaa8033f29790fbb05bc7d32da5d2dba9fb04b59dc3b518f2d2179ee912b"}
Jul 07 02:18:48.199600 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:48.199429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-x9f6g" event={"ID":"3ab3d260-6260-4cd4-9400-43a9bef9179b","Type":"ContainerStarted","Data":"f8cf486c51330e78469c7ba6160ec4e2a073a3befa2fb84dce4b2efd0212343a"}
Jul 07 02:18:48.227579 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:48.227518    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-x9f6g" podStartSLOduration=2.227498728 podStartE2EDuration="2.227498728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:48.226340005 +0000 UTC m=+63.885448768" watchObservedRunningTime="2026-07-07 02:18:48.227498728 +0000 UTC m=+63.886607491"
Jul 07 02:18:48.447434 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:48.447324    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-br5cd_61893ef3-a907-4929-9848-e0fee89fd5d9/serve-healthcheck-canary/0.log"
Jul 07 02:18:49.049266 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:49.049241    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-n9t4p_c7544dc0-bff1-42bd-b00d-a8911eee438f/migrator/0.log"
Jul 07 02:18:49.246956 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:49.246921    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-n9t4p_c7544dc0-bff1-42bd-b00d-a8911eee438f/graceful-termination/0.log"
Jul 07 02:18:50.033371 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.033333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:50.033546 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.033391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:18:50.033762 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.033749    2566 scope.go:117] "RemoveContainer" containerID="0226e81644e05dadc8491a46aa3fe6b656e9ac6e139ca329687763ddcaf1fa4e"
Jul 07 02:18:50.033917 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:50.033898    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-bktlx_openshift-console-operator(4b6fc827-c7ca-4b38-8aef-63d8b12166bd)\"" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" podUID="4b6fc827-c7ca-4b38-8aef-63d8b12166bd"
Jul 07 02:18:50.506075 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.506034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:18:50.506525 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.506088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:50.508665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.508644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:18:50.508766 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.508644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:18:50.516977 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:50.516939    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:18:50.517132 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:50.517016    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs podName:eb530956-7b82-4d88-975f-3c3e042b7fb8 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:54.516993701 +0000 UTC m=+130.176102442 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs") pod "network-metrics-daemon-v2khl" (UID: "eb530956-7b82-4d88-975f-3c3e042b7fb8") : secret "metrics-daemon-secret" not found
Jul 07 02:18:50.519315 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.519285    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3cee6f8-980e-4530-b615-b02786a8e117-original-pull-secret\") pod \"global-pull-secret-syncer-gvjbd\" (UID: \"c3cee6f8-980e-4530-b615-b02786a8e117\") " pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:50.607332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.607280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:50.609960 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.609937    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:18:50.620556 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.620532    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:18:50.631271 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.631243    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5727\" (UniqueName: \"kubernetes.io/projected/bd3f6020-1ea4-4a96-bc08-7d370557b27d-kube-api-access-d5727\") pod \"network-check-target-ns6nx\" (UID: \"bd3f6020-1ea4-4a96-bc08-7d370557b27d\") " pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:50.666904 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.666867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hf7w2\""
Jul 07 02:18:50.667931 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.667913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvjbd"
Jul 07 02:18:50.673417 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.673394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:50.807524 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.807458    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ns6nx"]
Jul 07 02:18:50.811017 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:50.810992    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3f6020_1ea4_4a96_bc08_7d370557b27d.slice/crio-2808b04df66d1873d3297d7d5eb9859f82b0af61e17b4cd40f0516cd4735b818 WatchSource:0}: Error finding container 2808b04df66d1873d3297d7d5eb9859f82b0af61e17b4cd40f0516cd4735b818: Status 404 returned error can't find the container with id 2808b04df66d1873d3297d7d5eb9859f82b0af61e17b4cd40f0516cd4735b818
Jul 07 02:18:50.822542 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:50.822519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gvjbd"]
Jul 07 02:18:50.825561 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:50.825541    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cee6f8_980e_4530_b615_b02786a8e117.slice/crio-2c075c1eab30619477c0cc8581bdcad73308b67b180fadfee693e8c8958ee5c8 WatchSource:0}: Error finding container 2c075c1eab30619477c0cc8581bdcad73308b67b180fadfee693e8c8958ee5c8: Status 404 returned error can't find the container with id 2c075c1eab30619477c0cc8581bdcad73308b67b180fadfee693e8c8958ee5c8
Jul 07 02:18:51.210613 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:51.210575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gvjbd" event={"ID":"c3cee6f8-980e-4530-b615-b02786a8e117","Type":"ContainerStarted","Data":"2c075c1eab30619477c0cc8581bdcad73308b67b180fadfee693e8c8958ee5c8"}
Jul 07 02:18:51.211694 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:51.211655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ns6nx" event={"ID":"bd3f6020-1ea4-4a96-bc08-7d370557b27d","Type":"ContainerStarted","Data":"2808b04df66d1873d3297d7d5eb9859f82b0af61e17b4cd40f0516cd4735b818"}
Jul 07 02:18:51.817601 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:51.817040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:51.817601 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:51.817124    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:51.817601 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:51.817218    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:18:51.817601 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:51.817232    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:51.817601 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:51.817283    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:59.817264809 +0000 UTC m=+75.476373552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : configmap "kube-rbac-proxy" not found
Jul 07 02:18:51.817601 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:51.817301    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls podName:bffe9839-af0d-4f05-8fe0-f2d8bb7fc446 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:59.817292099 +0000 UTC m=+75.476400843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swmdm" (UID: "bffe9839-af0d-4f05-8fe0-f2d8bb7fc446") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:18:52.524386 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:52.524344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:52.527516 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:52.527490    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c43f4893-fdbb-422f-af88-799cb1e33ec6-metrics-tls\") pod \"dns-default-wttkz\" (UID: \"c43f4893-fdbb-422f-af88-799cb1e33ec6\") " pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:52.773740 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:52.773709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc8t9\""
Jul 07 02:18:52.782005 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:52.781914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:52.959840 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:52.959782    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wttkz"]
Jul 07 02:18:52.962734 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:18:52.962703    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43f4893_fdbb_422f_af88_799cb1e33ec6.slice/crio-02378a8f225e7fdc935efb6ba4fcbdb163088108d2ec4ebc25b762c1a310f52d WatchSource:0}: Error finding container 02378a8f225e7fdc935efb6ba4fcbdb163088108d2ec4ebc25b762c1a310f52d: Status 404 returned error can't find the container with id 02378a8f225e7fdc935efb6ba4fcbdb163088108d2ec4ebc25b762c1a310f52d
Jul 07 02:18:53.221928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:53.221851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wttkz" event={"ID":"c43f4893-fdbb-422f-af88-799cb1e33ec6","Type":"ContainerStarted","Data":"02378a8f225e7fdc935efb6ba4fcbdb163088108d2ec4ebc25b762c1a310f52d"}
Jul 07 02:18:55.452457 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:55.452414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:18:55.452898 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:55.452545    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:55.452898 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:18:55.452615    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls podName:4753d121-c72d-45fa-a249-ef8ccc73d525 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:11.452598425 +0000 UTC m=+87.111707168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-5pqn5" (UID: "4753d121-c72d-45fa-a249-ef8ccc73d525") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:18:57.237657 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.237613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gvjbd" event={"ID":"c3cee6f8-980e-4530-b615-b02786a8e117","Type":"ContainerStarted","Data":"cd4a3acb72692d6ab2033675c80be100829bf5dd55aeb37cdd0a4bdec41a2b84"}
Jul 07 02:18:57.239383 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.239352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wttkz" event={"ID":"c43f4893-fdbb-422f-af88-799cb1e33ec6","Type":"ContainerStarted","Data":"77f5eb4c443aeaef363a55a4172a0e8e2ba5c23f7535d6afbab2b52fd0824938"}
Jul 07 02:18:57.239493 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.239398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wttkz" event={"ID":"c43f4893-fdbb-422f-af88-799cb1e33ec6","Type":"ContainerStarted","Data":"2f3f1a29123eedfcfcfaece7f76725c12b27a4b65e3089ac57c1522f57fc46c0"}
Jul 07 02:18:57.239493 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.239433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wttkz"
Jul 07 02:18:57.240996 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.240971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ns6nx" event={"ID":"bd3f6020-1ea4-4a96-bc08-7d370557b27d","Type":"ContainerStarted","Data":"7dfba448688b87c2361f8cbe14c320310b4545e0d8a68ec79cc1957c331c0fbe"}
Jul 07 02:18:57.241113 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.241086    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:18:57.253009 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.252966    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gvjbd" podStartSLOduration=66.888314758 podStartE2EDuration="1m12.252950964s" totalImagesPullingTime="5.364636206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:57.252114539 +0000 UTC m=+72.911223300" watchObservedRunningTime="2026-07-07 02:18:57.252950964 +0000 UTC m=+72.912059726"
Jul 07 02:18:57.271447 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.271396    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wttkz" podStartSLOduration=34.039795013 podStartE2EDuration="37.271376231s" totalImagesPullingTime="3.231581218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:57.270515047 +0000 UTC m=+72.929623810" watchObservedRunningTime="2026-07-07 02:18:57.271376231 +0000 UTC m=+72.930484993"
Jul 07 02:18:57.287490 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:57.287437    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ns6nx" podStartSLOduration=67.908912779 podStartE2EDuration="1m13.287418437s" totalImagesPullingTime="5.378505658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:57.285446843 +0000 UTC m=+72.944555598" watchObservedRunningTime="2026-07-07 02:18:57.287418437 +0000 UTC m=+72.946527199"
Jul 07 02:18:59.891119 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:59.891068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:59.891119 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:59.891119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:59.891578 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:59.891561    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:59.893328 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:59.893291    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bffe9839-af0d-4f05-8fe0-f2d8bb7fc446-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swmdm\" (UID: \"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446\") " pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:18:59.988214 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:59.988186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vqqxg\""
Jul 07 02:18:59.996627 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:18:59.996609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-swmdm"
Jul 07 02:19:00.143338 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:00.143254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-swmdm"]
Jul 07 02:19:00.146403 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:00.146377    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffe9839_af0d_4f05_8fe0_f2d8bb7fc446.slice/crio-8ffd26d2e5fd4a04061c5a47497f298c8aaaed26361ffb7a4f687d155b3938cc WatchSource:0}: Error finding container 8ffd26d2e5fd4a04061c5a47497f298c8aaaed26361ffb7a4f687d155b3938cc: Status 404 returned error can't find the container with id 8ffd26d2e5fd4a04061c5a47497f298c8aaaed26361ffb7a4f687d155b3938cc
Jul 07 02:19:00.251428 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:00.251392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swmdm" event={"ID":"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446","Type":"ContainerStarted","Data":"8ffd26d2e5fd4a04061c5a47497f298c8aaaed26361ffb7a4f687d155b3938cc"}
Jul 07 02:19:01.843377 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:01.843348    2566 scope.go:117] "RemoveContainer" containerID="0226e81644e05dadc8491a46aa3fe6b656e9ac6e139ca329687763ddcaf1fa4e"
Jul 07 02:19:04.267832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:04.267806    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:19:04.268298 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:04.267908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" event={"ID":"4b6fc827-c7ca-4b38-8aef-63d8b12166bd","Type":"ContainerStarted","Data":"5a65206c5deb99385fc612009b5b17105df02038752ff56e19e5a099deea3bba"}
Jul 07 02:19:04.268298 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:04.268222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:19:04.269456 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:04.269425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swmdm" event={"ID":"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446","Type":"ContainerStarted","Data":"433326d63189c271041a737fe4bda2723d16e782470e5f4013d35734aecff6b5"}
Jul 07 02:19:04.272889 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:04.272863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-bktlx"
Jul 07 02:19:04.285007 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:04.284948    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-bktlx" podStartSLOduration=22.305886437 podStartE2EDuration="25.284929013s" totalImagesPullingTime="2.979042576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:04.284261823 +0000 UTC m=+79.943370587" watchObservedRunningTime="2026-07-07 02:19:04.284929013 +0000 UTC m=+79.944037767"
Jul 07 02:19:05.275415 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:05.275376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swmdm" event={"ID":"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446","Type":"ContainerStarted","Data":"e7c050997e122f18c23a9025b7d2e5813a5445dfd4add887993ff9763ef75be6"}
Jul 07 02:19:07.246919 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:07.246889    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wttkz"
Jul 07 02:19:07.283223 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:07.283181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swmdm" event={"ID":"bffe9839-af0d-4f05-8fe0-f2d8bb7fc446","Type":"ContainerStarted","Data":"73dc14f80e332cbd43daa851bb0dca796d8aa6e9b56bca164f65e282561e626e"}
Jul 07 02:19:07.301183 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:07.301129    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-swmdm" podStartSLOduration=16.682954554 podStartE2EDuration="23.301112536s" totalImagesPullingTime="6.618157982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:07.299963947 +0000 UTC m=+82.959072709" watchObservedRunningTime="2026-07-07 02:19:07.301112536 +0000 UTC m=+82.960221296"
Jul 07 02:19:10.343391 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.343355    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"]
Jul 07 02:19:10.345689 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.345651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.350517 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.350496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 02:19:10.350646 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.350552    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bm9c4\""
Jul 07 02:19:10.350646 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.350623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 02:19:10.351438 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.351418    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 02:19:10.351540 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.351453    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 02:19:10.361618 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.361594    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"]
Jul 07 02:19:10.478289 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.478252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a9f89d3-84af-49fb-a933-4e1ee44cb87a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-646f98fd48-2ct6t\" (UID: \"5a9f89d3-84af-49fb-a933-4e1ee44cb87a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.478482 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.478395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfjtk\" (UniqueName: \"kubernetes.io/projected/5a9f89d3-84af-49fb-a933-4e1ee44cb87a-kube-api-access-rfjtk\") pod \"managed-serviceaccount-addon-agent-646f98fd48-2ct6t\" (UID: \"5a9f89d3-84af-49fb-a933-4e1ee44cb87a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.505718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.505685    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-565bf886d4-98gzv"]
Jul 07 02:19:10.507777 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.507752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.511726 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.511699    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 02:19:10.511969 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.511953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 02:19:10.512061 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.511967    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 02:19:10.512357 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.512344    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lwxwm\""
Jul 07 02:19:10.518267 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.518248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 02:19:10.526412 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.526389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565bf886d4-98gzv"]
Jul 07 02:19:10.579137 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfjtk\" (UniqueName: \"kubernetes.io/projected/5a9f89d3-84af-49fb-a933-4e1ee44cb87a-kube-api-access-rfjtk\") pod \"managed-serviceaccount-addon-agent-646f98fd48-2ct6t\" (UID: \"5a9f89d3-84af-49fb-a933-4e1ee44cb87a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.579341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d47l\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-kube-api-access-4d47l\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-bound-sa-token\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfe3208a-0da7-497d-af10-07265cca435e-installation-pull-secrets\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a9f89d3-84af-49fb-a933-4e1ee44cb87a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-646f98fd48-2ct6t\" (UID: \"5a9f89d3-84af-49fb-a933-4e1ee44cb87a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.579341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfe3208a-0da7-497d-af10-07265cca435e-ca-trust-extracted\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579554 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfe3208a-0da7-497d-af10-07265cca435e-image-registry-private-configuration\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579554 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-registry-tls\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579554 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579402    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfe3208a-0da7-497d-af10-07265cca435e-registry-certificates\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.579554 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.579425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3208a-0da7-497d-af10-07265cca435e-trusted-ca\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.581796 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.581775    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a9f89d3-84af-49fb-a933-4e1ee44cb87a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-646f98fd48-2ct6t\" (UID: \"5a9f89d3-84af-49fb-a933-4e1ee44cb87a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.591785 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.591757    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfjtk\" (UniqueName: \"kubernetes.io/projected/5a9f89d3-84af-49fb-a933-4e1ee44cb87a-kube-api-access-rfjtk\") pod \"managed-serviceaccount-addon-agent-646f98fd48-2ct6t\" (UID: \"5a9f89d3-84af-49fb-a933-4e1ee44cb87a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.659838 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.659740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"
Jul 07 02:19:10.680245 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d47l\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-kube-api-access-4d47l\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680394 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-bound-sa-token\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680394 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfe3208a-0da7-497d-af10-07265cca435e-installation-pull-secrets\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680394 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfe3208a-0da7-497d-af10-07265cca435e-ca-trust-extracted\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680394 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfe3208a-0da7-497d-af10-07265cca435e-image-registry-private-configuration\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-registry-tls\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfe3208a-0da7-497d-af10-07265cca435e-registry-certificates\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3208a-0da7-497d-af10-07265cca435e-trusted-ca\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.680943 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.680916    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dfe3208a-0da7-497d-af10-07265cca435e-ca-trust-extracted\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.682275 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.681432    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dfe3208a-0da7-497d-af10-07265cca435e-registry-certificates\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.682937 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.682869    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dfe3208a-0da7-497d-af10-07265cca435e-trusted-ca\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.683163 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.683143    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dfe3208a-0da7-497d-af10-07265cca435e-installation-pull-secrets\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.683555 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.683533    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dfe3208a-0da7-497d-af10-07265cca435e-image-registry-private-configuration\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.683596 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.683577    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-registry-tls\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.690396 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.690370    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-bound-sa-token\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.690764 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.690746    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d47l\" (UniqueName: \"kubernetes.io/projected/dfe3208a-0da7-497d-af10-07265cca435e-kube-api-access-4d47l\") pod \"image-registry-565bf886d4-98gzv\" (UID: \"dfe3208a-0da7-497d-af10-07265cca435e\") " pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.783318 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.783297    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t"]
Jul 07 02:19:10.785660 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:10.785633    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a9f89d3_84af_49fb_a933_4e1ee44cb87a.slice/crio-334dbfd976c53e446d06fe4e817f8f4b0015a2d0ef2b03cf68768da638c174fb WatchSource:0}: Error finding container 334dbfd976c53e446d06fe4e817f8f4b0015a2d0ef2b03cf68768da638c174fb: Status 404 returned error can't find the container with id 334dbfd976c53e446d06fe4e817f8f4b0015a2d0ef2b03cf68768da638c174fb
Jul 07 02:19:10.817171 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.817134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:10.946576 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:10.944700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565bf886d4-98gzv"]
Jul 07 02:19:11.297554 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.297520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565bf886d4-98gzv" event={"ID":"dfe3208a-0da7-497d-af10-07265cca435e","Type":"ContainerStarted","Data":"835760472d6f2b34e16947b4322881bfed6421e5d6a9b00cbbbd16e07560a797"}
Jul 07 02:19:11.297732 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.297575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565bf886d4-98gzv" event={"ID":"dfe3208a-0da7-497d-af10-07265cca435e","Type":"ContainerStarted","Data":"5ddc212caa94fd27e4782639acfd65b740a9b1440d483bce20872c8e22f6e8c7"}
Jul 07 02:19:11.297732 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.297642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:11.298709 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.298686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t" event={"ID":"5a9f89d3-84af-49fb-a933-4e1ee44cb87a","Type":"ContainerStarted","Data":"334dbfd976c53e446d06fe4e817f8f4b0015a2d0ef2b03cf68768da638c174fb"}
Jul 07 02:19:11.320749 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.320702    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-565bf886d4-98gzv" podStartSLOduration=1.320662831 podStartE2EDuration="1.320662831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:11.320029589 +0000 UTC m=+86.979138363" watchObservedRunningTime="2026-07-07 02:19:11.320662831 +0000 UTC m=+86.979771593"
Jul 07 02:19:11.487903 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.487859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:19:11.490304 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.490280    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4753d121-c72d-45fa-a249-ef8ccc73d525-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-5pqn5\" (UID: \"4753d121-c72d-45fa-a249-ef8ccc73d525\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:19:11.731652 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.731620    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vs224\""
Jul 07 02:19:11.739703 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.739686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"
Jul 07 02:19:11.861185 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:11.861157    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5"]
Jul 07 02:19:11.864157 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:11.864125    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4753d121_c72d_45fa_a249_ef8ccc73d525.slice/crio-c5c3b068b8f9e61516219fa9a62254cbe30c58997c5ad1468fad3584d1cc94e3 WatchSource:0}: Error finding container c5c3b068b8f9e61516219fa9a62254cbe30c58997c5ad1468fad3584d1cc94e3: Status 404 returned error can't find the container with id c5c3b068b8f9e61516219fa9a62254cbe30c58997c5ad1468fad3584d1cc94e3
Jul 07 02:19:12.303705 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:12.303661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5" event={"ID":"4753d121-c72d-45fa-a249-ef8ccc73d525","Type":"ContainerStarted","Data":"c5c3b068b8f9e61516219fa9a62254cbe30c58997c5ad1468fad3584d1cc94e3"}
Jul 07 02:19:14.313114 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:14.313080    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5" event={"ID":"4753d121-c72d-45fa-a249-ef8ccc73d525","Type":"ContainerStarted","Data":"56a9d7e5603c081327180550b4a2bc920515b4712c1c76e4e240960f5a028572"}
Jul 07 02:19:14.332612 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:14.332566    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-5pqn5" podStartSLOduration=33.492208648 podStartE2EDuration="35.332552147s" totalImagesPullingTime="1.840343499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:14.331050838 +0000 UTC m=+89.990159599" watchObservedRunningTime="2026-07-07 02:19:14.332552147 +0000 UTC m=+89.991660887"
Jul 07 02:19:18.284495 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.284458    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"]
Jul 07 02:19:18.319754 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.319704    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"]
Jul 07 02:19:18.319923 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.319819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.323722 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.323696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 02:19:18.323878 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.323721    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 02:19:18.323878 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.323761    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ssdkq\""
Jul 07 02:19:18.323991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.323922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 02:19:18.440894 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.440856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.441094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.440926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.441094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.441004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.441094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.441056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztk4\" (UniqueName: \"kubernetes.io/projected/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-kube-api-access-jztk4\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.541742 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.541650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.541742 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.541709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztk4\" (UniqueName: \"kubernetes.io/projected/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-kube-api-access-jztk4\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.541906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.541745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.541906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.541788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.542426 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.542404    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.544091 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.544067    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.544196 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.544142    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.551002 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.550983    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztk4\" (UniqueName: \"kubernetes.io/projected/8baa35ce-c17b-4f0c-8b99-620fa133a0d6-kube-api-access-jztk4\") pod \"prometheus-operator-567f7b8b7c-cb6t9\" (UID: \"8baa35ce-c17b-4f0c-8b99-620fa133a0d6\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.629046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.629029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"
Jul 07 02:19:18.749648 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:18.749620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9"]
Jul 07 02:19:18.752449 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:18.752418    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8baa35ce_c17b_4f0c_8b99_620fa133a0d6.slice/crio-3542b4be6213efcadee3090bfcbb9f31dc839bf7a70da33c5eb6a27d21cc56ce WatchSource:0}: Error finding container 3542b4be6213efcadee3090bfcbb9f31dc839bf7a70da33c5eb6a27d21cc56ce: Status 404 returned error can't find the container with id 3542b4be6213efcadee3090bfcbb9f31dc839bf7a70da33c5eb6a27d21cc56ce
Jul 07 02:19:19.330844 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:19.330807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9" event={"ID":"8baa35ce-c17b-4f0c-8b99-620fa133a0d6","Type":"ContainerStarted","Data":"3542b4be6213efcadee3090bfcbb9f31dc839bf7a70da33c5eb6a27d21cc56ce"}
Jul 07 02:19:20.337894 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:20.337598    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9" event={"ID":"8baa35ce-c17b-4f0c-8b99-620fa133a0d6","Type":"ContainerStarted","Data":"dedf290f2918d44ea58351e87279a0a0ea0843070250d4908a9d6a5a61762aca"}
Jul 07 02:19:20.337894 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:20.337714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9" event={"ID":"8baa35ce-c17b-4f0c-8b99-620fa133a0d6","Type":"ContainerStarted","Data":"d2f0b837b3a65bbf76ff45af050ed9021e30de2d902c020d20294b1c07e47f75"}
Jul 07 02:19:20.358585 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:20.358526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-cb6t9" podStartSLOduration=1.214729131 podStartE2EDuration="2.358505372s" totalImagesPullingTime="1.143776241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:20.357959858 +0000 UTC m=+96.017068621" watchObservedRunningTime="2026-07-07 02:19:20.358505372 +0000 UTC m=+96.017614136"
Jul 07 02:19:22.347245 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.347204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t" event={"ID":"5a9f89d3-84af-49fb-a933-4e1ee44cb87a","Type":"ContainerStarted","Data":"a14744f97bda952bc51e438cec659f11d46da3f5969880f0f746ea4df5382ffc"}
Jul 07 02:19:22.369968 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.369913    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-646f98fd48-2ct6t" podStartSLOduration=1.499313203 podStartE2EDuration="12.369895865s" totalImagesPullingTime="10.870582662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:22.368654809 +0000 UTC m=+98.027763571" watchObservedRunningTime="2026-07-07 02:19:22.369895865 +0000 UTC m=+98.029004627"
Jul 07 02:19:22.620639 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.620550    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"]
Jul 07 02:19:22.623648 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.623057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.625525 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.625504    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 02:19:22.625659 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.625536    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 02:19:22.625836 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.625817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nt7kw\""
Jul 07 02:19:22.626741 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.626718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 02:19:22.638057 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.638037    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"]
Jul 07 02:19:22.656105 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.656083    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s9hxd"]
Jul 07 02:19:22.658262 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.658243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.661180 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.661156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 02:19:22.661823 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.661432    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 02:19:22.661823 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.661722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwsnb\""
Jul 07 02:19:22.661985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.661920    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 02:19:22.781835 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.781788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10710ff-0a3e-41fd-b942-a7146fec1cc3-metrics-client-ca\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.781995 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.781842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-wtmp\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.781995 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.781873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.781995 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.781899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-tls\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.781995 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.781937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq44x\" (UniqueName: \"kubernetes.io/projected/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-api-access-vq44x\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.781995 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.781972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.782256 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-textfile\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.782256 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.782256 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.782256 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-sys\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.782256 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.782438 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.782438 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.782438 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjm2\" (UniqueName: \"kubernetes.io/projected/c10710ff-0a3e-41fd-b942-a7146fec1cc3-kube-api-access-qrjm2\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.782438 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.782381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-root\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.883481 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10710ff-0a3e-41fd-b942-a7146fec1cc3-metrics-client-ca\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.883481 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-wtmp\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.883481 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883572    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-wtmp\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-tls\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq44x\" (UniqueName: \"kubernetes.io/projected/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-api-access-vq44x\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:19:22.883657    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-textfile\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.883792 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:19:22.883748    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-tls podName:cd612176-f5ee-4c27-b8b1-f2c00b9d5807 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:23.383726816 +0000 UTC m=+99.042835562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-n2tjj" (UID: "cd612176-f5ee-4c27-b8b1-f2c00b9d5807") : secret "kube-state-metrics-tls" not found
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-sys\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.883985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884005    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-textfile\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjm2\" (UniqueName: \"kubernetes.io/projected/c10710ff-0a3e-41fd-b942-a7146fec1cc3-kube-api-access-qrjm2\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-root\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884054    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-sys\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884147 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884088    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10710ff-0a3e-41fd-b942-a7146fec1cc3-metrics-client-ca\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884730 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884392    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c10710ff-0a3e-41fd-b942-a7146fec1cc3-root\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884730 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884579    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.884875 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884852    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.884954 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.884904    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.885049 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.885029    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.887074 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.887022    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-tls\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.887186 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.887104    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.887186 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.887104    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c10710ff-0a3e-41fd-b942-a7146fec1cc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.891817 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.891799    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjm2\" (UniqueName: \"kubernetes.io/projected/c10710ff-0a3e-41fd-b942-a7146fec1cc3-kube-api-access-qrjm2\") pod \"node-exporter-s9hxd\" (UID: \"c10710ff-0a3e-41fd-b942-a7146fec1cc3\") " pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.892465 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.892445    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq44x\" (UniqueName: \"kubernetes.io/projected/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-api-access-vq44x\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:22.970527 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:22.970489    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s9hxd"
Jul 07 02:19:22.980956 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:22.980932    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10710ff_0a3e_41fd_b942_a7146fec1cc3.slice/crio-52d926f5ae6022237fb74c27bfe13ace24003e93d945cb3e23dde20b40299df0 WatchSource:0}: Error finding container 52d926f5ae6022237fb74c27bfe13ace24003e93d945cb3e23dde20b40299df0: Status 404 returned error can't find the container with id 52d926f5ae6022237fb74c27bfe13ace24003e93d945cb3e23dde20b40299df0
Jul 07 02:19:23.352017 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:23.351977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9hxd" event={"ID":"c10710ff-0a3e-41fd-b942-a7146fec1cc3","Type":"ContainerStarted","Data":"52d926f5ae6022237fb74c27bfe13ace24003e93d945cb3e23dde20b40299df0"}
Jul 07 02:19:23.388812 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:23.388779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:23.391436 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:23.391377    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd612176-f5ee-4c27-b8b1-f2c00b9d5807-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-n2tjj\" (UID: \"cd612176-f5ee-4c27-b8b1-f2c00b9d5807\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:23.535719 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:23.535687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"
Jul 07 02:19:23.716970 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:23.716070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj"]
Jul 07 02:19:23.891310 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:23.891238    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd612176_f5ee_4c27_b8b1_f2c00b9d5807.slice/crio-2ce27cd7ce8a187f38f7b689a5100a2169df26b69ed108d5f5706b64510bd710 WatchSource:0}: Error finding container 2ce27cd7ce8a187f38f7b689a5100a2169df26b69ed108d5f5706b64510bd710: Status 404 returned error can't find the container with id 2ce27cd7ce8a187f38f7b689a5100a2169df26b69ed108d5f5706b64510bd710
Jul 07 02:19:24.358752 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:24.358698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj" event={"ID":"cd612176-f5ee-4c27-b8b1-f2c00b9d5807","Type":"ContainerStarted","Data":"2ce27cd7ce8a187f38f7b689a5100a2169df26b69ed108d5f5706b64510bd710"}
Jul 07 02:19:24.360896 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:24.360868    2566 generic.go:356] "Generic (PLEG): container finished" podID="c10710ff-0a3e-41fd-b942-a7146fec1cc3" containerID="5664011bd7b345d00cf9a4b43f1666a730883218210cedb2a1fe61ad5470e7d2" exitCode=0
Jul 07 02:19:24.361044 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:24.360947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9hxd" event={"ID":"c10710ff-0a3e-41fd-b942-a7146fec1cc3","Type":"ContainerDied","Data":"5664011bd7b345d00cf9a4b43f1666a730883218210cedb2a1fe61ad5470e7d2"}
Jul 07 02:19:25.367014 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:25.366978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9hxd" event={"ID":"c10710ff-0a3e-41fd-b942-a7146fec1cc3","Type":"ContainerStarted","Data":"963816cadcbad2918229750c8bb0722404891e799a75c07b2c7ae8e6ba40d9e5"}
Jul 07 02:19:25.367480 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:25.367043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s9hxd" event={"ID":"c10710ff-0a3e-41fd-b942-a7146fec1cc3","Type":"ContainerStarted","Data":"1b7a898a8eb8b6cb10538c15a3308f732505c59d765b52b3e6d09966296ccdaa"}
Jul 07 02:19:26.373074 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:26.373034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj" event={"ID":"cd612176-f5ee-4c27-b8b1-f2c00b9d5807","Type":"ContainerStarted","Data":"be87d6470a67aef2023982d993e43b258fb4c8b2114e68d768134feb0f5ff9f8"}
Jul 07 02:19:26.373481 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:26.373096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj" event={"ID":"cd612176-f5ee-4c27-b8b1-f2c00b9d5807","Type":"ContainerStarted","Data":"7776167cb35ad2752f23d55e82edd041c90df23766b9804fcba9722f8bd85a2d"}
Jul 07 02:19:26.373481 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:26.373115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj" event={"ID":"cd612176-f5ee-4c27-b8b1-f2c00b9d5807","Type":"ContainerStarted","Data":"9749b819710d126c9b1f9017bc198bcbd1119ee876abd52147aaa80a782d24e4"}
Jul 07 02:19:26.396267 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:26.396203    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-n2tjj" podStartSLOduration=2.64253757 podStartE2EDuration="4.396182119s" totalImagesPullingTime="1.753644549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:26.395124476 +0000 UTC m=+102.054233261" watchObservedRunningTime="2026-07-07 02:19:26.396182119 +0000 UTC m=+102.055290882"
Jul 07 02:19:26.396796 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:26.396767    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s9hxd" podStartSLOduration=3.354403474 podStartE2EDuration="4.39675735s" totalImagesPullingTime="968.803219ms" totalInitContainerRuntime="73.550657ms" isStatefulPod=true podCreationTimestamp="2026-07-07 02:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:25.388997694 +0000 UTC m=+101.048106455" watchObservedRunningTime="2026-07-07 02:19:26.39675735 +0000 UTC m=+102.055866114"
Jul 07 02:19:27.003598 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.003563    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74c499c889-vcbt9"]
Jul 07 02:19:27.019666 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.019628    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74c499c889-vcbt9"]
Jul 07 02:19:27.019845 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.019773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.026734 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.026700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 02:19:27.026903 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.026834    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 02:19:27.027242 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.027063    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jz6ng\""
Jul 07 02:19:27.027242 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.027105    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5osisavpe8753\""
Jul 07 02:19:27.027355 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.027300    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 02:19:27.027454 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.027431    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 02:19:27.122302 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/69503df1-69f1-4822-825d-8178b665c64f-metrics-server-audit-profiles\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.122302 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/69503df1-69f1-4822-825d-8178b665c64f-audit-log\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.122531 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-secret-metrics-server-client-certs\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.122531 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gllbg\" (UniqueName: \"kubernetes.io/projected/69503df1-69f1-4822-825d-8178b665c64f-kube-api-access-gllbg\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.122531 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-client-ca-bundle\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.122531 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69503df1-69f1-4822-825d-8178b665c64f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.122531 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.122506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-secret-metrics-server-tls\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.223814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/69503df1-69f1-4822-825d-8178b665c64f-metrics-server-audit-profiles\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/69503df1-69f1-4822-825d-8178b665c64f-audit-log\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-secret-metrics-server-client-certs\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gllbg\" (UniqueName: \"kubernetes.io/projected/69503df1-69f1-4822-825d-8178b665c64f-kube-api-access-gllbg\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-client-ca-bundle\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224008 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69503df1-69f1-4822-825d-8178b665c64f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224267 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.223984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-secret-metrics-server-tls\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224347 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.224321    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/69503df1-69f1-4822-825d-8178b665c64f-audit-log\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224853 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.224831    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69503df1-69f1-4822-825d-8178b665c64f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.224957 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.224847    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/69503df1-69f1-4822-825d-8178b665c64f-metrics-server-audit-profiles\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.226500 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.226469    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-secret-metrics-server-tls\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.226993 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.226972    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-client-ca-bundle\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.227073 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.227018    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/69503df1-69f1-4822-825d-8178b665c64f-secret-metrics-server-client-certs\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.233052 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.233022    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gllbg\" (UniqueName: \"kubernetes.io/projected/69503df1-69f1-4822-825d-8178b665c64f-kube-api-access-gllbg\") pod \"metrics-server-74c499c889-vcbt9\" (UID: \"69503df1-69f1-4822-825d-8178b665c64f\") " pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.331296 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.331211    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:27.403801 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.403766    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"]
Jul 07 02:19:27.426911 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.426792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"]
Jul 07 02:19:27.427086 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.426893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.429781 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.429756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tdpn8\""
Jul 07 02:19:27.429914 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.429902    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 07 02:19:27.512238 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.512196    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74c499c889-vcbt9"]
Jul 07 02:19:27.514303 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:27.514276    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69503df1_69f1_4822_825d_8178b665c64f.slice/crio-6862b5d0e86be640cd6896e1d263890184666ec13ff2a7e422ffa5aa35603b5a WatchSource:0}: Error finding container 6862b5d0e86be640cd6896e1d263890184666ec13ff2a7e422ffa5aa35603b5a: Status 404 returned error can't find the container with id 6862b5d0e86be640cd6896e1d263890184666ec13ff2a7e422ffa5aa35603b5a
Jul 07 02:19:27.528660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.528640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3bb8f20-0be7-4bfd-8319-86d6007c9b4d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4dsr9\" (UID: \"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.528852 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.528832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckmhs\" (UniqueName: \"kubernetes.io/projected/f3bb8f20-0be7-4bfd-8319-86d6007c9b4d-kube-api-access-ckmhs\") pod \"monitoring-plugin-d5dcff7cf-4dsr9\" (UID: \"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.629825 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.629665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckmhs\" (UniqueName: \"kubernetes.io/projected/f3bb8f20-0be7-4bfd-8319-86d6007c9b4d-kube-api-access-ckmhs\") pod \"monitoring-plugin-d5dcff7cf-4dsr9\" (UID: \"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.630009 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.629820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3bb8f20-0be7-4bfd-8319-86d6007c9b4d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4dsr9\" (UID: \"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.632258 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.632223    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f3bb8f20-0be7-4bfd-8319-86d6007c9b4d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-4dsr9\" (UID: \"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.640433 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.640409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckmhs\" (UniqueName: \"kubernetes.io/projected/f3bb8f20-0be7-4bfd-8319-86d6007c9b4d-kube-api-access-ckmhs\") pod \"monitoring-plugin-d5dcff7cf-4dsr9\" (UID: \"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.742769 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.742732    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:27.905938 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:27.905843    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"]
Jul 07 02:19:27.909571 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:27.909542    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bb8f20_0be7_4bfd_8319_86d6007c9b4d.slice/crio-5bd9388b36d6d3392cab905265b0782945522912eda6b25b7a01c0f607d7bfcb WatchSource:0}: Error finding container 5bd9388b36d6d3392cab905265b0782945522912eda6b25b7a01c0f607d7bfcb: Status 404 returned error can't find the container with id 5bd9388b36d6d3392cab905265b0782945522912eda6b25b7a01c0f607d7bfcb
Jul 07 02:19:28.247453 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:28.247419    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ns6nx"
Jul 07 02:19:28.383562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:28.383521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9" event={"ID":"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d","Type":"ContainerStarted","Data":"5bd9388b36d6d3392cab905265b0782945522912eda6b25b7a01c0f607d7bfcb"}
Jul 07 02:19:28.384786 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:28.384762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9" event={"ID":"69503df1-69f1-4822-825d-8178b665c64f","Type":"ContainerStarted","Data":"6862b5d0e86be640cd6896e1d263890184666ec13ff2a7e422ffa5aa35603b5a"}
Jul 07 02:19:30.395937 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:30.395895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9" event={"ID":"69503df1-69f1-4822-825d-8178b665c64f","Type":"ContainerStarted","Data":"c2507e59266483ecd4d59b920fc778622ce7aac67b18d20aa7c642e951ce2f64"}
Jul 07 02:19:30.397458 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:30.397432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9" event={"ID":"f3bb8f20-0be7-4bfd-8319-86d6007c9b4d","Type":"ContainerStarted","Data":"d994ab6476b3a3c5868a1328f69e8cacdaae8582f93f232046841db3e7ae3b68"}
Jul 07 02:19:30.397551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:30.397480    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:30.402358 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:30.402335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9"
Jul 07 02:19:30.415707 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:30.415634    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9" podStartSLOduration=2.799715259 podStartE2EDuration="4.415620242s" totalImagesPullingTime="1.615904983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:30.414850131 +0000 UTC m=+106.073958894" watchObservedRunningTime="2026-07-07 02:19:30.415620242 +0000 UTC m=+106.074729004"
Jul 07 02:19:30.432076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:30.432024    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-4dsr9" podStartSLOduration=1.78635493 podStartE2EDuration="3.432003062s" totalImagesPullingTime="1.645648132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:30.431795117 +0000 UTC m=+106.090903879" watchObservedRunningTime="2026-07-07 02:19:30.432003062 +0000 UTC m=+106.091111824"
Jul 07 02:19:32.307877 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:32.307846    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-565bf886d4-98gzv"
Jul 07 02:19:47.331793 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:47.331755    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:47.332272 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:47.331853    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:19:54.487295 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.487258    2566 generic.go:356] "Generic (PLEG): container finished" podID="f975e7d9-c1a7-41bd-89fb-6fe6f706aff6" containerID="17f7f8d85d23341ac54749de6848345f5fdf0d455260fac219d7a643d7d15d4c" exitCode=0
Jul 07 02:19:54.487753 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.487305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh" event={"ID":"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6","Type":"ContainerDied","Data":"17f7f8d85d23341ac54749de6848345f5fdf0d455260fac219d7a643d7d15d4c"}
Jul 07 02:19:54.487753 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.487617    2566 scope.go:117] "RemoveContainer" containerID="17f7f8d85d23341ac54749de6848345f5fdf0d455260fac219d7a643d7d15d4c"
Jul 07 02:19:54.557871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.557845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:19:54.559981 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.559958    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb530956-7b82-4d88-975f-3c3e042b7fb8-metrics-certs\") pod \"network-metrics-daemon-v2khl\" (UID: \"eb530956-7b82-4d88-975f-3c3e042b7fb8\") " pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:19:54.857964 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.857891    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ggshj\""
Jul 07 02:19:54.865925 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:54.865901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2khl"
Jul 07 02:19:55.003487 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:55.000151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2khl"]
Jul 07 02:19:55.008643 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:19:55.006868    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb530956_7b82_4d88_975f_3c3e042b7fb8.slice/crio-761fdd7b40014d93ee750152d936720563ba15ac7fda9301719d22e32938311b WatchSource:0}: Error finding container 761fdd7b40014d93ee750152d936720563ba15ac7fda9301719d22e32938311b: Status 404 returned error can't find the container with id 761fdd7b40014d93ee750152d936720563ba15ac7fda9301719d22e32938311b
Jul 07 02:19:55.493482 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:55.493442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-4mqhh" event={"ID":"f975e7d9-c1a7-41bd-89fb-6fe6f706aff6","Type":"ContainerStarted","Data":"e07a1784d00f566527d424b083a6917637568b1abae1bf9b9cb0b6eb24eab8f6"}
Jul 07 02:19:55.494804 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:55.494765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2khl" event={"ID":"eb530956-7b82-4d88-975f-3c3e042b7fb8","Type":"ContainerStarted","Data":"761fdd7b40014d93ee750152d936720563ba15ac7fda9301719d22e32938311b"}
Jul 07 02:19:56.500634 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:56.500601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2khl" event={"ID":"eb530956-7b82-4d88-975f-3c3e042b7fb8","Type":"ContainerStarted","Data":"8d0a9cc2e0be968430a9e8e202bb4f3fb5e918f27ea4cc3e585c160eee00b793"}
Jul 07 02:19:56.501121 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:56.500658    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2khl" event={"ID":"eb530956-7b82-4d88-975f-3c3e042b7fb8","Type":"ContainerStarted","Data":"fc034fc71fe2b11cccb973aee86337ae11dbe5039ee8552ebdf29b85b8bbafd6"}
Jul 07 02:19:56.518918 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:19:56.518870    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v2khl" podStartSLOduration=131.53105445 podStartE2EDuration="2m12.518853398s" totalImagesPullingTime="987.798948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:56.517856605 +0000 UTC m=+132.176965366" watchObservedRunningTime="2026-07-07 02:19:56.518853398 +0000 UTC m=+132.177962344"
Jul 07 02:20:07.341276 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:20:07.341240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:20:07.346568 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:20:07.346511    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74c499c889-vcbt9"
Jul 07 02:20:09.554350 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:20:09.554314    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d4475f0-c6bb-4743-9419-a2bb4df52d56" containerID="960ac7c676022830cd08237ab7bb1e069682f7acd1c1ec28ddd12629d21e16d0" exitCode=0
Jul 07 02:20:09.554744 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:20:09.554387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh" event={"ID":"0d4475f0-c6bb-4743-9419-a2bb4df52d56","Type":"ContainerDied","Data":"960ac7c676022830cd08237ab7bb1e069682f7acd1c1ec28ddd12629d21e16d0"}
Jul 07 02:20:09.554744 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:20:09.554715    2566 scope.go:117] "RemoveContainer" containerID="960ac7c676022830cd08237ab7bb1e069682f7acd1c1ec28ddd12629d21e16d0"
Jul 07 02:20:10.561625 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:20:10.561591    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-78fnh" event={"ID":"0d4475f0-c6bb-4743-9419-a2bb4df52d56","Type":"ContainerStarted","Data":"fd62bbbe575ed15593efe9ccfe30d4ce64db950349787b8a2d47c2936cf7a915"}
Jul 07 02:22:44.710896 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:22:44.710858    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:22:44.716236 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:22:44.716209    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:22:44.718818 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:22:44.718799    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:22:44.722163 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:22:44.722141    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 02:22:44.725225 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:22:44.725203    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:23:45.752658 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.752579    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-vq9z6"]
Jul 07 02:23:45.755902 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.755878    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:45.758728 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.758705    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7cjsm\""
Jul 07 02:23:45.760465 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.760448    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 02:23:45.760985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.760969    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 07 02:23:45.761059 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.760974    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 02:23:45.768515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.768495    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-vq9z6"]
Jul 07 02:23:45.817028 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.816998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb628\" (UniqueName: \"kubernetes.io/projected/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-kube-api-access-jb628\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:45.817179 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.817037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:45.918057 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.918016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb628\" (UniqueName: \"kubernetes.io/projected/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-kube-api-access-jb628\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:45.918231 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.918065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:45.918278 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:23:45.918246    2566 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 07 02:23:45.918340 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:23:45.918328    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert podName:3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2 nodeName:}" failed. No retries permitted until 2026-07-07 02:23:46.418306939 +0000 UTC m=+362.077415684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert") pod "kserve-controller-manager-f8f7445f4-vq9z6" (UID: "3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2") : secret "kserve-webhook-server-cert" not found
Jul 07 02:23:45.926960 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:45.926929    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb628\" (UniqueName: \"kubernetes.io/projected/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-kube-api-access-jb628\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:46.422094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:46.422061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:46.424408 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:46.424383    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert\") pod \"kserve-controller-manager-f8f7445f4-vq9z6\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") " pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:46.667809 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:46.667767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:46.795777 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:46.795752    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-vq9z6"]
Jul 07 02:23:46.798136 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:23:46.798107    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af2bc1d_fee6_4385_a39f_d4f11b7aa1f2.slice/crio-bc5f93aa123a69ee429f6d0647905b9749bcd384e0979a6a1610ab365b8a31db WatchSource:0}: Error finding container bc5f93aa123a69ee429f6d0647905b9749bcd384e0979a6a1610ab365b8a31db: Status 404 returned error can't find the container with id bc5f93aa123a69ee429f6d0647905b9749bcd384e0979a6a1610ab365b8a31db
Jul 07 02:23:46.799657 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:46.799638    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:23:47.350267 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:47.350234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6" event={"ID":"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2","Type":"ContainerStarted","Data":"bc5f93aa123a69ee429f6d0647905b9749bcd384e0979a6a1610ab365b8a31db"}
Jul 07 02:23:50.363857 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:50.363810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6" event={"ID":"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2","Type":"ContainerStarted","Data":"a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f"}
Jul 07 02:23:50.364352 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:50.363912    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:23:50.382157 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:23:50.382104    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6" podStartSLOduration=2.748786233 podStartE2EDuration="5.38208875s" totalImagesPullingTime="2.633302517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:50.380920005 +0000 UTC m=+366.040028769" watchObservedRunningTime="2026-07-07 02:23:50.38208875 +0000 UTC m=+366.041197515"
Jul 07 02:24:21.372701 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:21.372654    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:24:22.892897 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:22.892854    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-vq9z6"]
Jul 07 02:24:22.893392 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:22.893042    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6" podUID="3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" containerName="manager" containerID="cri-o://a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f" gracePeriod=10
Jul 07 02:24:22.927406 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:22.927367    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-94lch"]
Jul 07 02:24:22.933842 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:22.933814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:22.940108 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:22.940075    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-94lch"]
Jul 07 02:24:23.025615 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.025582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbef2b15-dc5b-41ee-b0ca-b16c490d2464-cert\") pod \"kserve-controller-manager-f8f7445f4-94lch\" (UID: \"dbef2b15-dc5b-41ee-b0ca-b16c490d2464\") " pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.025782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.025624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wscl\" (UniqueName: \"kubernetes.io/projected/dbef2b15-dc5b-41ee-b0ca-b16c490d2464-kube-api-access-2wscl\") pod \"kserve-controller-manager-f8f7445f4-94lch\" (UID: \"dbef2b15-dc5b-41ee-b0ca-b16c490d2464\") " pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.126203 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.126162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbef2b15-dc5b-41ee-b0ca-b16c490d2464-cert\") pod \"kserve-controller-manager-f8f7445f4-94lch\" (UID: \"dbef2b15-dc5b-41ee-b0ca-b16c490d2464\") " pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.126376 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.126219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wscl\" (UniqueName: \"kubernetes.io/projected/dbef2b15-dc5b-41ee-b0ca-b16c490d2464-kube-api-access-2wscl\") pod \"kserve-controller-manager-f8f7445f4-94lch\" (UID: \"dbef2b15-dc5b-41ee-b0ca-b16c490d2464\") " pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.128440 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.128412    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbef2b15-dc5b-41ee-b0ca-b16c490d2464-cert\") pod \"kserve-controller-manager-f8f7445f4-94lch\" (UID: \"dbef2b15-dc5b-41ee-b0ca-b16c490d2464\") " pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.136472 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.136449    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wscl\" (UniqueName: \"kubernetes.io/projected/dbef2b15-dc5b-41ee-b0ca-b16c490d2464-kube-api-access-2wscl\") pod \"kserve-controller-manager-f8f7445f4-94lch\" (UID: \"dbef2b15-dc5b-41ee-b0ca-b16c490d2464\") " pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.175128 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.175048    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:24:23.276082 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.276044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:23.327478 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.327439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert\") pod \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") "
Jul 07 02:24:23.327690 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.327518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb628\" (UniqueName: \"kubernetes.io/projected/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-kube-api-access-jb628\") pod \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\" (UID: \"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2\") "
Jul 07 02:24:23.329816 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.329783    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-kube-api-access-jb628" (OuterVolumeSpecName: "kube-api-access-jb628") pod "3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" (UID: "3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2"). InnerVolumeSpecName "kube-api-access-jb628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:24:23.329816 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.329802    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert" (OuterVolumeSpecName: "cert") pod "3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" (UID: "3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:23.421144 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.421110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-94lch"]
Jul 07 02:24:23.424326 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:24:23.424297    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbef2b15_dc5b_41ee_b0ca_b16c490d2464.slice/crio-ce9e0abe4091fd764d2b606d81e4cb8c480348bde0d54bfa000680782d3d3f28 WatchSource:0}: Error finding container ce9e0abe4091fd764d2b606d81e4cb8c480348bde0d54bfa000680782d3d3f28: Status 404 returned error can't find the container with id ce9e0abe4091fd764d2b606d81e4cb8c480348bde0d54bfa000680782d3d3f28
Jul 07 02:24:23.428212 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.428194    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:24:23.428277 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.428215    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb628\" (UniqueName: \"kubernetes.io/projected/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2-kube-api-access-jb628\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:24:23.486038 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.486007    2566 generic.go:356] "Generic (PLEG): container finished" podID="3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" containerID="a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f" exitCode=0
Jul 07 02:24:23.486230 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.486060    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6"
Jul 07 02:24:23.486230 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.486085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6" event={"ID":"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2","Type":"ContainerDied","Data":"a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f"}
Jul 07 02:24:23.486230 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.486139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-vq9z6" event={"ID":"3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2","Type":"ContainerDied","Data":"bc5f93aa123a69ee429f6d0647905b9749bcd384e0979a6a1610ab365b8a31db"}
Jul 07 02:24:23.486230 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.486170    2566 scope.go:117] "RemoveContainer" containerID="a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f"
Jul 07 02:24:23.487331 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.487230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-94lch" event={"ID":"dbef2b15-dc5b-41ee-b0ca-b16c490d2464","Type":"ContainerStarted","Data":"ce9e0abe4091fd764d2b606d81e4cb8c480348bde0d54bfa000680782d3d3f28"}
Jul 07 02:24:23.494622 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.494599    2566 scope.go:117] "RemoveContainer" containerID="a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f"
Jul 07 02:24:23.494922 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:24:23.494901    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f\": container with ID starting with a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f not found: ID does not exist" containerID="a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f"
Jul 07 02:24:23.495014 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.494929    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f"} err="failed to get container status \"a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f\": rpc error: code = NotFound desc = could not find container \"a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f\": container with ID starting with a15d7c1064563dbe4fd98c8269df9c91ffc1427612dff122355cae1b6fe6842f not found: ID does not exist"
Jul 07 02:24:23.508638 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.508617    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-vq9z6"]
Jul 07 02:24:23.515121 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:23.515099    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-vq9z6"]
Jul 07 02:24:24.493276 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:24.493242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-94lch" event={"ID":"dbef2b15-dc5b-41ee-b0ca-b16c490d2464","Type":"ContainerStarted","Data":"e5b4135f86fa03b825c05c694855d773a0e17b07d06ed1120ffe13562aa9620a"}
Jul 07 02:24:24.493653 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:24.493307    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:24.514621 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:24.513723    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8f7445f4-94lch" podStartSLOduration=2.19780551 podStartE2EDuration="2.513702148s" totalImagesPullingTime="315.896638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:24.512549304 +0000 UTC m=+400.171658091" watchObservedRunningTime="2026-07-07 02:24:24.513702148 +0000 UTC m=+400.172810911"
Jul 07 02:24:24.847553 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:24.847457    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" path="/var/lib/kubelet/pods/3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2/volumes"
Jul 07 02:24:55.501331 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:55.501301    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8f7445f4-94lch"
Jul 07 02:24:56.555164 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.555127    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-bbncc"]
Jul 07 02:24:56.555616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.555586    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" containerName="manager"
Jul 07 02:24:56.555616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.555604    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" containerName="manager"
Jul 07 02:24:56.555751 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.555693    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af2bc1d-fee6-4385-a39f-d4f11b7aa1f2" containerName="manager"
Jul 07 02:24:56.558566 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.558544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.563422 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.563399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 02:24:56.563548 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.563459    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-5hx55\""
Jul 07 02:24:56.566728 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.566698    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-swrch"]
Jul 07 02:24:56.568718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.568703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:56.571647 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.571626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-65djc\""
Jul 07 02:24:56.571775 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.571625    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 02:24:56.574625 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.574595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-bbncc"]
Jul 07 02:24:56.583021 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.582989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c74d5a-5437-4fee-98e2-ca2ed8116469-tls-certs\") pod \"model-serving-api-fd65d7d6-bbncc\" (UID: \"78c74d5a-5437-4fee-98e2-ca2ed8116469\") " pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.583120 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.583084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b9a881b-14bf-478a-ac7e-71eb3213a547-cert\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:56.583171 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.583125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmfd\" (UniqueName: \"kubernetes.io/projected/2b9a881b-14bf-478a-ac7e-71eb3213a547-kube-api-access-fnmfd\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:56.583171 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.583150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9zh\" (UniqueName: \"kubernetes.io/projected/78c74d5a-5437-4fee-98e2-ca2ed8116469-kube-api-access-lq9zh\") pod \"model-serving-api-fd65d7d6-bbncc\" (UID: \"78c74d5a-5437-4fee-98e2-ca2ed8116469\") " pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.597775 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.597750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-swrch"]
Jul 07 02:24:56.684285 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.684248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b9a881b-14bf-478a-ac7e-71eb3213a547-cert\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:56.684503 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.684297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmfd\" (UniqueName: \"kubernetes.io/projected/2b9a881b-14bf-478a-ac7e-71eb3213a547-kube-api-access-fnmfd\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:56.684503 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.684319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9zh\" (UniqueName: \"kubernetes.io/projected/78c74d5a-5437-4fee-98e2-ca2ed8116469-kube-api-access-lq9zh\") pod \"model-serving-api-fd65d7d6-bbncc\" (UID: \"78c74d5a-5437-4fee-98e2-ca2ed8116469\") " pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.684503 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.684351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c74d5a-5437-4fee-98e2-ca2ed8116469-tls-certs\") pod \"model-serving-api-fd65d7d6-bbncc\" (UID: \"78c74d5a-5437-4fee-98e2-ca2ed8116469\") " pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.684503 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:24:56.684482    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 02:24:56.684754 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:24:56.684552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b9a881b-14bf-478a-ac7e-71eb3213a547-cert podName:2b9a881b-14bf-478a-ac7e-71eb3213a547 nodeName:}" failed. No retries permitted until 2026-07-07 02:24:57.184532496 +0000 UTC m=+432.843641240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2b9a881b-14bf-478a-ac7e-71eb3213a547-cert") pod "odh-model-controller-84b54b7f97-swrch" (UID: "2b9a881b-14bf-478a-ac7e-71eb3213a547") : secret "odh-model-controller-webhook-cert" not found
Jul 07 02:24:56.686784 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.686762    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c74d5a-5437-4fee-98e2-ca2ed8116469-tls-certs\") pod \"model-serving-api-fd65d7d6-bbncc\" (UID: \"78c74d5a-5437-4fee-98e2-ca2ed8116469\") " pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.696170 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.696147    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9zh\" (UniqueName: \"kubernetes.io/projected/78c74d5a-5437-4fee-98e2-ca2ed8116469-kube-api-access-lq9zh\") pod \"model-serving-api-fd65d7d6-bbncc\" (UID: \"78c74d5a-5437-4fee-98e2-ca2ed8116469\") " pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:56.696313 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.696211    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmfd\" (UniqueName: \"kubernetes.io/projected/2b9a881b-14bf-478a-ac7e-71eb3213a547-kube-api-access-fnmfd\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:56.871581 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:56.871490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:57.009159 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:57.009126    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-bbncc"]
Jul 07 02:24:57.012232 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:24:57.012206    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c74d5a_5437_4fee_98e2_ca2ed8116469.slice/crio-48b0f49fd6890a493eeddba4033b2ea36b77ceefdf0260474aa581129ce21d98 WatchSource:0}: Error finding container 48b0f49fd6890a493eeddba4033b2ea36b77ceefdf0260474aa581129ce21d98: Status 404 returned error can't find the container with id 48b0f49fd6890a493eeddba4033b2ea36b77ceefdf0260474aa581129ce21d98
Jul 07 02:24:57.189735 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:57.189623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b9a881b-14bf-478a-ac7e-71eb3213a547-cert\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:57.191999 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:57.191978    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b9a881b-14bf-478a-ac7e-71eb3213a547-cert\") pod \"odh-model-controller-84b54b7f97-swrch\" (UID: \"2b9a881b-14bf-478a-ac7e-71eb3213a547\") " pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:57.481143 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:57.481106    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:24:57.610655 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:57.610560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-swrch"]
Jul 07 02:24:57.613817 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:24:57.613777    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9a881b_14bf_478a_ac7e_71eb3213a547.slice/crio-35ee23868761b2aac256e053c7a87f5a909ded69fc788369da182f161da2a74a WatchSource:0}: Error finding container 35ee23868761b2aac256e053c7a87f5a909ded69fc788369da182f161da2a74a: Status 404 returned error can't find the container with id 35ee23868761b2aac256e053c7a87f5a909ded69fc788369da182f161da2a74a
Jul 07 02:24:57.619080 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:57.619049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-bbncc" event={"ID":"78c74d5a-5437-4fee-98e2-ca2ed8116469","Type":"ContainerStarted","Data":"48b0f49fd6890a493eeddba4033b2ea36b77ceefdf0260474aa581129ce21d98"}
Jul 07 02:24:58.626407 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:58.626358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-swrch" event={"ID":"2b9a881b-14bf-478a-ac7e-71eb3213a547","Type":"ContainerStarted","Data":"35ee23868761b2aac256e053c7a87f5a909ded69fc788369da182f161da2a74a"}
Jul 07 02:24:59.632563 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:59.632527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-bbncc" event={"ID":"78c74d5a-5437-4fee-98e2-ca2ed8116469","Type":"ContainerStarted","Data":"00c02d47cbec14abd36a2d949efeb75ee94e0c30b241be43049d68098df1541f"}
Jul 07 02:24:59.633053 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:59.632689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:24:59.650722 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:24:59.650656    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-bbncc" podStartSLOduration=1.997700507 podStartE2EDuration="3.650640964s" totalImagesPullingTime="1.652940457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:59.648797034 +0000 UTC m=+435.307905800" watchObservedRunningTime="2026-07-07 02:24:59.650640964 +0000 UTC m=+435.309749729"
Jul 07 02:25:00.638734 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:00.638614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-swrch" event={"ID":"2b9a881b-14bf-478a-ac7e-71eb3213a547","Type":"ContainerStarted","Data":"5dfe8e1052c53f005f96595583943875cad10ff84e06579de2d2b9e8d165307e"}
Jul 07 02:25:00.639128 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:00.638801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:25:00.656356 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:00.656295    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-swrch" podStartSLOduration=2.00692599 podStartE2EDuration="4.656273137s" totalImagesPullingTime="2.649347147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:25:00.655395597 +0000 UTC m=+436.314504358" watchObservedRunningTime="2026-07-07 02:25:00.656273137 +0000 UTC m=+436.315381902"
Jul 07 02:25:10.643057 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:10.642982    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-bbncc"
Jul 07 02:25:11.644805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:11.644775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-swrch"
Jul 07 02:25:12.470656 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.470621    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5f4kr"]
Jul 07 02:25:12.472974 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.472957    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.475390 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.475361    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 02:25:12.475527 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.475400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-49tjq\""
Jul 07 02:25:12.481686 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.481644    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5f4kr"]
Jul 07 02:25:12.520757 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.520721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvj9\" (UniqueName: \"kubernetes.io/projected/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-kube-api-access-gpvj9\") pod \"s3-init-5f4kr\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") " pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.520925 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.520785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-model-cache\") pod \"s3-init-5f4kr\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") " pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.622115 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.622077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-model-cache\") pod \"s3-init-5f4kr\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") " pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.622286 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.622148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvj9\" (UniqueName: \"kubernetes.io/projected/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-kube-api-access-gpvj9\") pod \"s3-init-5f4kr\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") " pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.622472 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.622453    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-model-cache\") pod \"s3-init-5f4kr\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") " pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.631057 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.631019    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvj9\" (UniqueName: \"kubernetes.io/projected/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-kube-api-access-gpvj9\") pod \"s3-init-5f4kr\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") " pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.782990 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.782896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5f4kr"
Jul 07 02:25:12.914316 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:12.914261    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5f4kr"]
Jul 07 02:25:12.916922 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:25:12.916885    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8725c2b4_4c3e_4c9d_a340_cd729e65b7de.slice/crio-2fe68f93dfd7c354f4b96e4442a5da1c07419705c2f190b38ab81ad43fb87874 WatchSource:0}: Error finding container 2fe68f93dfd7c354f4b96e4442a5da1c07419705c2f190b38ab81ad43fb87874: Status 404 returned error can't find the container with id 2fe68f93dfd7c354f4b96e4442a5da1c07419705c2f190b38ab81ad43fb87874
Jul 07 02:25:13.690188 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:13.690146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5f4kr" event={"ID":"8725c2b4-4c3e-4c9d-a340-cd729e65b7de","Type":"ContainerStarted","Data":"2fe68f93dfd7c354f4b96e4442a5da1c07419705c2f190b38ab81ad43fb87874"}
Jul 07 02:25:17.708145 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:17.708062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5f4kr" event={"ID":"8725c2b4-4c3e-4c9d-a340-cd729e65b7de","Type":"ContainerStarted","Data":"90f7162cec4abbd7c74312fdf85770f2b0e7155c3458052acdd4c4d2b2669d09"}
Jul 07 02:25:19.718091 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:19.718053    2566 generic.go:356] "Generic (PLEG): container finished" podID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerID="90f7162cec4abbd7c74312fdf85770f2b0e7155c3458052acdd4c4d2b2669d09" exitCode=0
Jul 07 02:25:19.718460 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:19.718122    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5f4kr" event={"ID":"8725c2b4-4c3e-4c9d-a340-cd729e65b7de","Type":"ContainerDied","Data":"90f7162cec4abbd7c74312fdf85770f2b0e7155c3458052acdd4c4d2b2669d09"}
Jul 07 02:25:24.740864 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:24.740812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5f4kr" event={"ID":"8725c2b4-4c3e-4c9d-a340-cd729e65b7de","Type":"ContainerStarted","Data":"b206c661dc1299646d3267533fa646be880e554c49fb1da2cd0a22399dc08ab0"}
Jul 07 02:25:24.762322 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:25:24.762272    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-5f4kr" podStartSLOduration=1.547745102 podStartE2EDuration="12.76225517s" totalImagesPullingTime="8.943577086s" totalInitContainerRuntime="2.270932982s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:25:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:25:24.761766711 +0000 UTC m=+460.420875474" watchObservedRunningTime="2026-07-07 02:25:24.76225517 +0000 UTC m=+460.421363933"
Jul 07 02:26:29.999425 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:29.999392    2566 generic.go:356] "Generic (PLEG): container finished" podID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerID="b206c661dc1299646d3267533fa646be880e554c49fb1da2cd0a22399dc08ab0" exitCode=0
Jul 07 02:26:29.999958 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:29.999468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5f4kr" event={"ID":"8725c2b4-4c3e-4c9d-a340-cd729e65b7de","Type":"ContainerDied","Data":"b206c661dc1299646d3267533fa646be880e554c49fb1da2cd0a22399dc08ab0"}
Jul 07 02:26:31.127241 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.127219    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5f4kr"
Jul 07 02:26:31.195845 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.195809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-model-cache\") pod \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") "
Jul 07 02:26:31.196028 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.195888    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpvj9\" (UniqueName: \"kubernetes.io/projected/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-kube-api-access-gpvj9\") pod \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\" (UID: \"8725c2b4-4c3e-4c9d-a340-cd729e65b7de\") "
Jul 07 02:26:31.198056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.198026    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-kube-api-access-gpvj9" (OuterVolumeSpecName: "kube-api-access-gpvj9") pod "8725c2b4-4c3e-4c9d-a340-cd729e65b7de" (UID: "8725c2b4-4c3e-4c9d-a340-cd729e65b7de"). InnerVolumeSpecName "kube-api-access-gpvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:26:31.214481 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.214450    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-model-cache" (OuterVolumeSpecName: "model-cache") pod "8725c2b4-4c3e-4c9d-a340-cd729e65b7de" (UID: "8725c2b4-4c3e-4c9d-a340-cd729e65b7de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:26:31.296396 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.296306    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-model-cache\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:26:31.296396 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:31.296336    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvj9\" (UniqueName: \"kubernetes.io/projected/8725c2b4-4c3e-4c9d-a340-cd729e65b7de-kube-api-access-gpvj9\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:26:32.007948 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:32.007916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5f4kr" event={"ID":"8725c2b4-4c3e-4c9d-a340-cd729e65b7de","Type":"ContainerDied","Data":"2fe68f93dfd7c354f4b96e4442a5da1c07419705c2f190b38ab81ad43fb87874"}
Jul 07 02:26:32.007948 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:32.007945    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5f4kr"
Jul 07 02:26:32.007948 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:32.007955    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe68f93dfd7c354f4b96e4442a5da1c07419705c2f190b38ab81ad43fb87874"
Jul 07 02:26:39.116052 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.115965    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"]
Jul 07 02:26:39.116489 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.116396    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerName="download-hf-model"
Jul 07 02:26:39.116489 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.116415    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerName="download-hf-model"
Jul 07 02:26:39.116489 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.116425    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerName="s3-init"
Jul 07 02:26:39.116489 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.116433    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerName="s3-init"
Jul 07 02:26:39.116713 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.116528    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8725c2b4-4c3e-4c9d-a340-cd729e65b7de" containerName="s3-init"
Jul 07 02:26:39.119398 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.119377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.126145 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.126123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-49tjq\""
Jul 07 02:26:39.126238 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.126159    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 07 02:26:39.126238 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.126209    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 07 02:26:39.132493 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.132472    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"]
Jul 07 02:26:39.157590 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.157558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/643c6f51-c488-4420-9236-3fa5ce33ed5e-data\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.157761 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.157603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ctf\" (UniqueName: \"kubernetes.io/projected/643c6f51-c488-4420-9236-3fa5ce33ed5e-kube-api-access-n9ctf\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.157761 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.157650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/643c6f51-c488-4420-9236-3fa5ce33ed5e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.258790 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.258759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/643c6f51-c488-4420-9236-3fa5ce33ed5e-data\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.258967 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.258807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ctf\" (UniqueName: \"kubernetes.io/projected/643c6f51-c488-4420-9236-3fa5ce33ed5e-kube-api-access-n9ctf\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.258967 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.258853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/643c6f51-c488-4420-9236-3fa5ce33ed5e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.259220 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.259197    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/643c6f51-c488-4420-9236-3fa5ce33ed5e-data\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.261168 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.261148    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/643c6f51-c488-4420-9236-3fa5ce33ed5e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.268137 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.268115    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ctf\" (UniqueName: \"kubernetes.io/projected/643c6f51-c488-4420-9236-3fa5ce33ed5e-kube-api-access-n9ctf\") pod \"seaweedfs-tls-custom-8b44d767d-nfpl8\" (UID: \"643c6f51-c488-4420-9236-3fa5ce33ed5e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.428501 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.428419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"
Jul 07 02:26:39.557019 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:39.556986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nfpl8"]
Jul 07 02:26:39.560764 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:26:39.560727    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643c6f51_c488_4420_9236_3fa5ce33ed5e.slice/crio-3759bc7b4105fed3d79f177575903b67b5dd8d19d488874339548653223a23a1 WatchSource:0}: Error finding container 3759bc7b4105fed3d79f177575903b67b5dd8d19d488874339548653223a23a1: Status 404 returned error can't find the container with id 3759bc7b4105fed3d79f177575903b67b5dd8d19d488874339548653223a23a1
Jul 07 02:26:40.039885 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:40.039847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8" event={"ID":"643c6f51-c488-4420-9236-3fa5ce33ed5e","Type":"ContainerStarted","Data":"3759bc7b4105fed3d79f177575903b67b5dd8d19d488874339548653223a23a1"}
Jul 07 02:26:43.053178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.053136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8" event={"ID":"643c6f51-c488-4420-9236-3fa5ce33ed5e","Type":"ContainerStarted","Data":"3c70441a2f0d36c3560f5b90b742491a38dbc745c3ec4f20ab34b84cb9f7f335"}
Jul 07 02:26:43.073188 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.073071    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-nfpl8" podStartSLOduration=0.812136382 podStartE2EDuration="4.073056849s" totalImagesPullingTime="3.260920467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:26:43.071811368 +0000 UTC m=+538.730920132" watchObservedRunningTime="2026-07-07 02:26:43.073056849 +0000 UTC m=+538.732165611"
Jul 07 02:26:43.389224 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.389144    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-ggq4g"]
Jul 07 02:26:43.392379 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.392356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.402699 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.402654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-ggq4g"]
Jul 07 02:26:43.497553 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.497499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010e7297-9c37-4087-b4f8-610459c09340-model-cache\") pod \"s3-tls-init-custom-ggq4g\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") " pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.497751 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.497572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt727\" (UniqueName: \"kubernetes.io/projected/010e7297-9c37-4087-b4f8-610459c09340-kube-api-access-pt727\") pod \"s3-tls-init-custom-ggq4g\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") " pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.598526 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.598477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt727\" (UniqueName: \"kubernetes.io/projected/010e7297-9c37-4087-b4f8-610459c09340-kube-api-access-pt727\") pod \"s3-tls-init-custom-ggq4g\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") " pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.598743 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.598559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010e7297-9c37-4087-b4f8-610459c09340-model-cache\") pod \"s3-tls-init-custom-ggq4g\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") " pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.598970 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.598952    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010e7297-9c37-4087-b4f8-610459c09340-model-cache\") pod \"s3-tls-init-custom-ggq4g\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") " pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.608519 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.608480    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt727\" (UniqueName: \"kubernetes.io/projected/010e7297-9c37-4087-b4f8-610459c09340-kube-api-access-pt727\") pod \"s3-tls-init-custom-ggq4g\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") " pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.700746 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.700631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:43.840203 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:43.840145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-ggq4g"]
Jul 07 02:26:43.842623 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:26:43.842592    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod010e7297_9c37_4087_b4f8_610459c09340.slice/crio-54128c59459b9d2851cadd318cfb894602c944ee6caa601c1e78a8d3ecc4ae7f WatchSource:0}: Error finding container 54128c59459b9d2851cadd318cfb894602c944ee6caa601c1e78a8d3ecc4ae7f: Status 404 returned error can't find the container with id 54128c59459b9d2851cadd318cfb894602c944ee6caa601c1e78a8d3ecc4ae7f
Jul 07 02:26:44.059431 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:44.059393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-ggq4g" event={"ID":"010e7297-9c37-4087-b4f8-610459c09340","Type":"ContainerStarted","Data":"aa1af375c04fef486b3ccadd7a7c555bbe0dd8f46a57f6cc2d9b52c9a254fce9"}
Jul 07 02:26:44.059431 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:44.059446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-ggq4g" event={"ID":"010e7297-9c37-4087-b4f8-610459c09340","Type":"ContainerStarted","Data":"54128c59459b9d2851cadd318cfb894602c944ee6caa601c1e78a8d3ecc4ae7f"}
Jul 07 02:26:46.070055 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:46.070018    2566 generic.go:356] "Generic (PLEG): container finished" podID="010e7297-9c37-4087-b4f8-610459c09340" containerID="aa1af375c04fef486b3ccadd7a7c555bbe0dd8f46a57f6cc2d9b52c9a254fce9" exitCode=0
Jul 07 02:26:46.070448 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:46.070088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-ggq4g" event={"ID":"010e7297-9c37-4087-b4f8-610459c09340","Type":"ContainerDied","Data":"aa1af375c04fef486b3ccadd7a7c555bbe0dd8f46a57f6cc2d9b52c9a254fce9"}
Jul 07 02:26:47.075826 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:47.075792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-ggq4g" event={"ID":"010e7297-9c37-4087-b4f8-610459c09340","Type":"ContainerStarted","Data":"375ab9866a039c86befbd2bc1e6c838397c84579cf2f64637c54ab221503b32c"}
Jul 07 02:26:47.097195 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:47.097116    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-ggq4g" podStartSLOduration=2.424748304 podStartE2EDuration="4.097099383s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.672351079s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:26:47.096045908 +0000 UTC m=+542.755154694" watchObservedRunningTime="2026-07-07 02:26:47.097099383 +0000 UTC m=+542.756208144"
Jul 07 02:26:54.105956 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:54.105923    2566 generic.go:356] "Generic (PLEG): container finished" podID="010e7297-9c37-4087-b4f8-610459c09340" containerID="375ab9866a039c86befbd2bc1e6c838397c84579cf2f64637c54ab221503b32c" exitCode=0
Jul 07 02:26:54.106307 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:54.105991    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-ggq4g" event={"ID":"010e7297-9c37-4087-b4f8-610459c09340","Type":"ContainerDied","Data":"375ab9866a039c86befbd2bc1e6c838397c84579cf2f64637c54ab221503b32c"}
Jul 07 02:26:55.256491 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.256460    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:55.305833 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.305800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010e7297-9c37-4087-b4f8-610459c09340-model-cache\") pod \"010e7297-9c37-4087-b4f8-610459c09340\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") "
Jul 07 02:26:55.305833 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.305839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt727\" (UniqueName: \"kubernetes.io/projected/010e7297-9c37-4087-b4f8-610459c09340-kube-api-access-pt727\") pod \"010e7297-9c37-4087-b4f8-610459c09340\" (UID: \"010e7297-9c37-4087-b4f8-610459c09340\") "
Jul 07 02:26:55.307903 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.307869    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010e7297-9c37-4087-b4f8-610459c09340-kube-api-access-pt727" (OuterVolumeSpecName: "kube-api-access-pt727") pod "010e7297-9c37-4087-b4f8-610459c09340" (UID: "010e7297-9c37-4087-b4f8-610459c09340"). InnerVolumeSpecName "kube-api-access-pt727". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:26:55.327762 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.327722    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/010e7297-9c37-4087-b4f8-610459c09340-model-cache" (OuterVolumeSpecName: "model-cache") pod "010e7297-9c37-4087-b4f8-610459c09340" (UID: "010e7297-9c37-4087-b4f8-610459c09340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:26:55.406482 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.406387    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/010e7297-9c37-4087-b4f8-610459c09340-model-cache\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:26:55.406482 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:55.406420    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt727\" (UniqueName: \"kubernetes.io/projected/010e7297-9c37-4087-b4f8-610459c09340-kube-api-access-pt727\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:26:56.114985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:56.114957    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-ggq4g"
Jul 07 02:26:56.114985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:56.114979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-ggq4g" event={"ID":"010e7297-9c37-4087-b4f8-610459c09340","Type":"ContainerDied","Data":"54128c59459b9d2851cadd318cfb894602c944ee6caa601c1e78a8d3ecc4ae7f"}
Jul 07 02:26:56.115182 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:56.115012    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54128c59459b9d2851cadd318cfb894602c944ee6caa601c1e78a8d3ecc4ae7f"
Jul 07 02:26:59.694618 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.694562    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-wxx59"]
Jul 07 02:26:59.695270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.695018    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010e7297-9c37-4087-b4f8-610459c09340" containerName="s3-tls-init-custom"
Jul 07 02:26:59.695270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.695036    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="010e7297-9c37-4087-b4f8-610459c09340" containerName="s3-tls-init-custom"
Jul 07 02:26:59.695270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.695068    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010e7297-9c37-4087-b4f8-610459c09340" containerName="download-hf-model"
Jul 07 02:26:59.695270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.695076    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="010e7297-9c37-4087-b4f8-610459c09340" containerName="download-hf-model"
Jul 07 02:26:59.695270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.695149    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="010e7297-9c37-4087-b4f8-610459c09340" containerName="s3-tls-init-custom"
Jul 07 02:26:59.713649 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.711044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-wxx59"]
Jul 07 02:26:59.713649 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.711160    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:26:59.714231 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.714202    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 07 02:26:59.843034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.842994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66df9d51-a062-4e90-a128-22f8e42e0cdb-model-cache\") pod \"s3-tls-init-serving-wxx59\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") " pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:26:59.843188 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.843059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqqrv\" (UniqueName: \"kubernetes.io/projected/66df9d51-a062-4e90-a128-22f8e42e0cdb-kube-api-access-dqqrv\") pod \"s3-tls-init-serving-wxx59\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") " pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:26:59.943975 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.943946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66df9d51-a062-4e90-a128-22f8e42e0cdb-model-cache\") pod \"s3-tls-init-serving-wxx59\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") " pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:26:59.944127 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.944029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqqrv\" (UniqueName: \"kubernetes.io/projected/66df9d51-a062-4e90-a128-22f8e42e0cdb-kube-api-access-dqqrv\") pod \"s3-tls-init-serving-wxx59\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") " pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:26:59.944397 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.944373    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66df9d51-a062-4e90-a128-22f8e42e0cdb-model-cache\") pod \"s3-tls-init-serving-wxx59\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") " pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:26:59.953336 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:26:59.953288    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqqrv\" (UniqueName: \"kubernetes.io/projected/66df9d51-a062-4e90-a128-22f8e42e0cdb-kube-api-access-dqqrv\") pod \"s3-tls-init-serving-wxx59\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") " pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:27:00.025778 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:00.025657    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:27:00.167827 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:00.167791    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-wxx59"]
Jul 07 02:27:00.170914 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:27:00.170886    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66df9d51_a062_4e90_a128_22f8e42e0cdb.slice/crio-afcbac26a6cd41a6b9f5f59c98ba85fcd34cf34e281b608a63019f6cedf8698b WatchSource:0}: Error finding container afcbac26a6cd41a6b9f5f59c98ba85fcd34cf34e281b608a63019f6cedf8698b: Status 404 returned error can't find the container with id afcbac26a6cd41a6b9f5f59c98ba85fcd34cf34e281b608a63019f6cedf8698b
Jul 07 02:27:01.138054 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:01.138008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wxx59" event={"ID":"66df9d51-a062-4e90-a128-22f8e42e0cdb","Type":"ContainerStarted","Data":"0bc944085666c56270af0500e9f0a076932740a87265dfed61ae95eedd765cf0"}
Jul 07 02:27:01.138485 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:01.138068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wxx59" event={"ID":"66df9d51-a062-4e90-a128-22f8e42e0cdb","Type":"ContainerStarted","Data":"afcbac26a6cd41a6b9f5f59c98ba85fcd34cf34e281b608a63019f6cedf8698b"}
Jul 07 02:27:02.143178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:02.143147    2566 generic.go:356] "Generic (PLEG): container finished" podID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerID="0bc944085666c56270af0500e9f0a076932740a87265dfed61ae95eedd765cf0" exitCode=0
Jul 07 02:27:02.143548 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:02.143227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wxx59" event={"ID":"66df9d51-a062-4e90-a128-22f8e42e0cdb","Type":"ContainerDied","Data":"0bc944085666c56270af0500e9f0a076932740a87265dfed61ae95eedd765cf0"}
Jul 07 02:27:03.149832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:03.149798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wxx59" event={"ID":"66df9d51-a062-4e90-a128-22f8e42e0cdb","Type":"ContainerStarted","Data":"442baf546209ccd2fe0056db292e011fb95d54ac9ea6cfe1626127fe969b2059"}
Jul 07 02:27:03.171018 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:03.170964    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-wxx59" podStartSLOduration=2.379430765 podStartE2EDuration="4.170949281s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.791518516s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:03.17057818 +0000 UTC m=+558.829686942" watchObservedRunningTime="2026-07-07 02:27:03.170949281 +0000 UTC m=+558.830058042"
Jul 07 02:27:09.180117 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:09.180083    2566 generic.go:356] "Generic (PLEG): container finished" podID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerID="442baf546209ccd2fe0056db292e011fb95d54ac9ea6cfe1626127fe969b2059" exitCode=0
Jul 07 02:27:09.180524 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:09.180155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wxx59" event={"ID":"66df9d51-a062-4e90-a128-22f8e42e0cdb","Type":"ContainerDied","Data":"442baf546209ccd2fe0056db292e011fb95d54ac9ea6cfe1626127fe969b2059"}
Jul 07 02:27:10.325291 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.325264    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:27:10.438739 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.438642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqqrv\" (UniqueName: \"kubernetes.io/projected/66df9d51-a062-4e90-a128-22f8e42e0cdb-kube-api-access-dqqrv\") pod \"66df9d51-a062-4e90-a128-22f8e42e0cdb\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") "
Jul 07 02:27:10.438739 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.438727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66df9d51-a062-4e90-a128-22f8e42e0cdb-model-cache\") pod \"66df9d51-a062-4e90-a128-22f8e42e0cdb\" (UID: \"66df9d51-a062-4e90-a128-22f8e42e0cdb\") "
Jul 07 02:27:10.440758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.440726    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66df9d51-a062-4e90-a128-22f8e42e0cdb-kube-api-access-dqqrv" (OuterVolumeSpecName: "kube-api-access-dqqrv") pod "66df9d51-a062-4e90-a128-22f8e42e0cdb" (UID: "66df9d51-a062-4e90-a128-22f8e42e0cdb"). InnerVolumeSpecName "kube-api-access-dqqrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:27:10.465067 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.465028    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df9d51-a062-4e90-a128-22f8e42e0cdb-model-cache" (OuterVolumeSpecName: "model-cache") pod "66df9d51-a062-4e90-a128-22f8e42e0cdb" (UID: "66df9d51-a062-4e90-a128-22f8e42e0cdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:27:10.539765 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.539715    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqqrv\" (UniqueName: \"kubernetes.io/projected/66df9d51-a062-4e90-a128-22f8e42e0cdb-kube-api-access-dqqrv\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:27:10.539765 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:10.539761    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66df9d51-a062-4e90-a128-22f8e42e0cdb-model-cache\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:27:11.189810 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:11.189729    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-wxx59"
Jul 07 02:27:11.189810 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:11.189737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-wxx59" event={"ID":"66df9d51-a062-4e90-a128-22f8e42e0cdb","Type":"ContainerDied","Data":"afcbac26a6cd41a6b9f5f59c98ba85fcd34cf34e281b608a63019f6cedf8698b"}
Jul 07 02:27:11.189810 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:11.189775    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afcbac26a6cd41a6b9f5f59c98ba85fcd34cf34e281b608a63019f6cedf8698b"
Jul 07 02:27:22.982261 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.982224    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"]
Jul 07 02:27:22.982655 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.982539    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerName="s3-tls-init-serving"
Jul 07 02:27:22.982655 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.982551    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerName="s3-tls-init-serving"
Jul 07 02:27:22.982655 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.982561    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerName="download-hf-model"
Jul 07 02:27:22.982655 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.982567    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerName="download-hf-model"
Jul 07 02:27:22.982655 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.982628    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="66df9d51-a062-4e90-a128-22f8e42e0cdb" containerName="s3-tls-init-serving"
Jul 07 02:27:22.985215 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.985198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:22.988641 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.988611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 07 02:27:22.988814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.988615    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:27:22.988814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.988692    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:27:22.988814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.988618    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 07 02:27:22.989116 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.989093    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:27:22.989116 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.989102    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:27:22.989289 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:22.989123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76zzm\""
Jul 07 02:27:23.001707 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.001654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"]
Jul 07 02:27:23.146360 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.146317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/298db630-4dd9-46b0-9bad-24d8347b5211-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.146360 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.146374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74n5\" (UniqueName: \"kubernetes.io/projected/298db630-4dd9-46b0-9bad-24d8347b5211-kube-api-access-z74n5\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.146616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.146403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.146616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.146449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.146616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.146501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.246983 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.246892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/298db630-4dd9-46b0-9bad-24d8347b5211-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.246983 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.246935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74n5\" (UniqueName: \"kubernetes.io/projected/298db630-4dd9-46b0-9bad-24d8347b5211-kube-api-access-z74n5\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.246983 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.246956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.247252 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.247001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.247252 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.247043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.247252 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:27:23.247210    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 07 02:27:23.247400 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:27:23.247295    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls podName:298db630-4dd9-46b0-9bad-24d8347b5211 nodeName:}" failed. No retries permitted until 2026-07-07 02:27:23.747273023 +0000 UTC m=+579.406381764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls") pod "isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" (UID: "298db630-4dd9-46b0-9bad-24d8347b5211") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 07 02:27:23.247400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.247321    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/298db630-4dd9-46b0-9bad-24d8347b5211-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.247595 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.247577    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.247752 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.247735    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.257762 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.257740    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74n5\" (UniqueName: \"kubernetes.io/projected/298db630-4dd9-46b0-9bad-24d8347b5211-kube-api-access-z74n5\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.752338 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.752299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.754991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.754962    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-699589c4f4-xrdkf\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:23.895886 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:23.895843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:24.042870 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:24.042839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"]
Jul 07 02:27:24.045760 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:27:24.045729    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298db630_4dd9_46b0_9bad_24d8347b5211.slice/crio-29bdc1587e3d21f413adc71e8af5b9508f205c804bdd7f52c19b48ed6a522046 WatchSource:0}: Error finding container 29bdc1587e3d21f413adc71e8af5b9508f205c804bdd7f52c19b48ed6a522046: Status 404 returned error can't find the container with id 29bdc1587e3d21f413adc71e8af5b9508f205c804bdd7f52c19b48ed6a522046
Jul 07 02:27:24.243726 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:24.243691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerStarted","Data":"4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1"}
Jul 07 02:27:24.243913 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:24.243759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerStarted","Data":"29bdc1587e3d21f413adc71e8af5b9508f205c804bdd7f52c19b48ed6a522046"}
Jul 07 02:27:28.263240 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:28.263205    2566 generic.go:356] "Generic (PLEG): container finished" podID="298db630-4dd9-46b0-9bad-24d8347b5211" containerID="4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1" exitCode=0
Jul 07 02:27:28.263693 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:28.263284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerDied","Data":"4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1"}
Jul 07 02:27:42.345007 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:42.344967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerStarted","Data":"f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd"}
Jul 07 02:27:44.742311 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:44.742282    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:27:44.743454 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:44.743433    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:27:44.749834 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:44.749813    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:27:44.750733 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:44.750714    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:27:45.362900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:45.362855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerStarted","Data":"823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e"}
Jul 07 02:27:47.374136 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.374095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerStarted","Data":"e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9"}
Jul 07 02:27:47.374549 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.374276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:47.374549 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.374340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:47.374549 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.374375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:47.375727 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.375663    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:27:47.376258 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.376237    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:27:47.397772 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:47.397602    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podStartSLOduration=3.2644792000000002 podStartE2EDuration="25.397584762s" totalImagesPullingTime="18.546236619s" totalInitContainerRuntime="3.586868943s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:27:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:47.396612625 +0000 UTC m=+603.055721387" watchObservedRunningTime="2026-07-07 02:27:47.397584762 +0000 UTC m=+603.056693524"
Jul 07 02:27:48.378400 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:48.378349    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:27:48.378885 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:48.378862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:27:53.384217 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:53.383467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:27:53.389032 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:53.388994    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:27:53.389477 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:27:53.389449    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:28:03.386331 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:03.386289    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:28:03.386818 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:03.386729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:28:13.386718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:13.386611    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:28:13.387178 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:13.387042    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:28:23.386116 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:23.386077    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:28:23.386562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:23.386536    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:28:33.385872 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:33.385824    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:28:33.386550 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:33.386525    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:28:43.385948 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:43.385904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:28:43.386488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:43.386463    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:28:53.385659 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:53.385622    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:28:53.386494 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:28:53.386463    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:03.386850 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:03.386820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:03.387409 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:03.387378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:18.063963 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.063930    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"]
Jul 07 02:29:18.064520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.064256    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" containerID="cri-o://f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd" gracePeriod=30
Jul 07 02:29:18.064520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.064437    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" containerID="cri-o://823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e" gracePeriod=30
Jul 07 02:29:18.064520 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.064453    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" containerID="cri-o://e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9" gracePeriod=30
Jul 07 02:29:18.272426 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.272385    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"]
Jul 07 02:29:18.276007 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.275988    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.279380 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.279355    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 07 02:29:18.280042 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.280025    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 07 02:29:18.295826 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.295746    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"]
Jul 07 02:29:18.336086 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.335996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.336086 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.336040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.336086 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.336059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bt8b\" (UniqueName: \"kubernetes.io/projected/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-api-access-8bt8b\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.336305 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.336116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.336305 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.336136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.379153 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.379111    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 02:29:18.437444 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.437405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.437444 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.437450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.437699 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.437472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bt8b\" (UniqueName: \"kubernetes.io/projected/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-api-access-8bt8b\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.437699 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.437514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.437699 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.437532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.438027 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.437991    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.438333 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.438313    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.438333 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.438323    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.439992 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.439973    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.446365 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.446338    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bt8b\" (UniqueName: \"kubernetes.io/projected/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-api-access-8bt8b\") pod \"isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.586358 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.586258    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:18.724507 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.724477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"]
Jul 07 02:29:18.726576 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:29:18.726548    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee872aa_f474_45fe_9f2a_7d0c937e99cb.slice/crio-51dd73af9bf9dd9705f4c187f4c7f097f95178fc04d27c1aa39e141d1562ffa4 WatchSource:0}: Error finding container 51dd73af9bf9dd9705f4c187f4c7f097f95178fc04d27c1aa39e141d1562ffa4: Status 404 returned error can't find the container with id 51dd73af9bf9dd9705f4c187f4c7f097f95178fc04d27c1aa39e141d1562ffa4
Jul 07 02:29:18.783926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.783893    2566 generic.go:356] "Generic (PLEG): container finished" podID="298db630-4dd9-46b0-9bad-24d8347b5211" containerID="823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e" exitCode=2
Jul 07 02:29:18.784038 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.783956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerDied","Data":"823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e"}
Jul 07 02:29:18.785715 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:18.785689    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerStarted","Data":"51dd73af9bf9dd9705f4c187f4c7f097f95178fc04d27c1aa39e141d1562ffa4"}
Jul 07 02:29:19.791068 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:19.791034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerStarted","Data":"38d20df642cd7199dc5a4f57d66b18eb6274390f38b72bbd8a31dbff7fefe997"}
Jul 07 02:29:22.805302 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:22.805266    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerID="38d20df642cd7199dc5a4f57d66b18eb6274390f38b72bbd8a31dbff7fefe997" exitCode=0
Jul 07 02:29:22.805710 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:22.805341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerDied","Data":"38d20df642cd7199dc5a4f57d66b18eb6274390f38b72bbd8a31dbff7fefe997"}
Jul 07 02:29:22.807971 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:22.807940    2566 generic.go:356] "Generic (PLEG): container finished" podID="298db630-4dd9-46b0-9bad-24d8347b5211" containerID="f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd" exitCode=0
Jul 07 02:29:22.808082 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:22.808030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerDied","Data":"f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd"}
Jul 07 02:29:23.378568 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.378514    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 02:29:23.386037 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.386004    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:29:23.388621 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.388594    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:23.814716 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.814666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerStarted","Data":"444ad6992a4dcaae5888f04f58f7ab52e2901f2a4fb1a3d0e10a546e326dc2bc"}
Jul 07 02:29:23.815201 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.814748    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerStarted","Data":"182bb6312f9cb5401bb6d10435922674d74bba8a4b708648a5ae42a52d63de07"}
Jul 07 02:29:23.815201 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.814772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerStarted","Data":"92b85a47c8c6d3347507543e3fb0cb4449006f85fe62c509ce9cd5a6daec15d7"}
Jul 07 02:29:23.815201 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.815058    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:23.815337 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.815211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:23.816583 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.816552    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:29:23.837288 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:23.837234    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podStartSLOduration=2.360485146 podStartE2EDuration="5.83722065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.476735504s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:23.834748454 +0000 UTC m=+699.493857218" watchObservedRunningTime="2026-07-07 02:29:23.83722065 +0000 UTC m=+699.496329411"
Jul 07 02:29:24.819868 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:24.819831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:24.820347 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:24.819930    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:29:24.821111 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:24.821079    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:25.824094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:25.824033    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:29:25.824566 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:25.824542    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:28.379441 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:28.379396    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 02:29:28.379887 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:28.379585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:30.829189 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:30.829155    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:29:30.829826 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:30.829786    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:29:30.830212 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:30.830185    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:33.378876 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:33.378828    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 02:29:33.385902 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:33.385862    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:29:33.388538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:33.388502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:38.378962 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:38.378861    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 02:29:40.830451 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:40.830408    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:29:40.830940 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:40.830885    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:43.378929 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:43.378888    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 07 02:29:43.386259 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:43.386218    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 07 02:29:43.386395 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:43.386378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:43.387752 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:43.387724    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:43.387832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:43.387823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:48.222832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.222806    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:48.399745 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.399628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z74n5\" (UniqueName: \"kubernetes.io/projected/298db630-4dd9-46b0-9bad-24d8347b5211-kube-api-access-z74n5\") pod \"298db630-4dd9-46b0-9bad-24d8347b5211\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") "
Jul 07 02:29:48.399745 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.399710    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-kube-rbac-proxy-sar-config\") pod \"298db630-4dd9-46b0-9bad-24d8347b5211\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") "
Jul 07 02:29:48.399745 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.399742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/298db630-4dd9-46b0-9bad-24d8347b5211-kserve-provision-location\") pod \"298db630-4dd9-46b0-9bad-24d8347b5211\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") "
Jul 07 02:29:48.400013 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.399780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-cabundle-cert\") pod \"298db630-4dd9-46b0-9bad-24d8347b5211\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") "
Jul 07 02:29:48.400013 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.399847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls\") pod \"298db630-4dd9-46b0-9bad-24d8347b5211\" (UID: \"298db630-4dd9-46b0-9bad-24d8347b5211\") "
Jul 07 02:29:48.400112 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.400024    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/298db630-4dd9-46b0-9bad-24d8347b5211-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "298db630-4dd9-46b0-9bad-24d8347b5211" (UID: "298db630-4dd9-46b0-9bad-24d8347b5211"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:29:48.400112 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.400080    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "298db630-4dd9-46b0-9bad-24d8347b5211" (UID: "298db630-4dd9-46b0-9bad-24d8347b5211"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:48.400186 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.400167    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "298db630-4dd9-46b0-9bad-24d8347b5211" (UID: "298db630-4dd9-46b0-9bad-24d8347b5211"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:29:48.401991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.401963    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298db630-4dd9-46b0-9bad-24d8347b5211-kube-api-access-z74n5" (OuterVolumeSpecName: "kube-api-access-z74n5") pod "298db630-4dd9-46b0-9bad-24d8347b5211" (UID: "298db630-4dd9-46b0-9bad-24d8347b5211"). InnerVolumeSpecName "kube-api-access-z74n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:29:48.402106 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.402043    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "298db630-4dd9-46b0-9bad-24d8347b5211" (UID: "298db630-4dd9-46b0-9bad-24d8347b5211"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:29:48.501278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.501238    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/298db630-4dd9-46b0-9bad-24d8347b5211-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:29:48.501278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.501272    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z74n5\" (UniqueName: \"kubernetes.io/projected/298db630-4dd9-46b0-9bad-24d8347b5211-kube-api-access-z74n5\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:29:48.501278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.501283    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:29:48.501278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.501292    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/298db630-4dd9-46b0-9bad-24d8347b5211-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:29:48.501538 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.501301    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/298db630-4dd9-46b0-9bad-24d8347b5211-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:29:48.932370 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.932339    2566 generic.go:356] "Generic (PLEG): container finished" podID="298db630-4dd9-46b0-9bad-24d8347b5211" containerID="e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9" exitCode=0
Jul 07 02:29:48.932512 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.932409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"
Jul 07 02:29:48.932512 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.932429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerDied","Data":"e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9"}
Jul 07 02:29:48.932512 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.932493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf" event={"ID":"298db630-4dd9-46b0-9bad-24d8347b5211","Type":"ContainerDied","Data":"29bdc1587e3d21f413adc71e8af5b9508f205c804bdd7f52c19b48ed6a522046"}
Jul 07 02:29:48.932611 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.932524    2566 scope.go:117] "RemoveContainer" containerID="e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9"
Jul 07 02:29:48.940563 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.940547    2566 scope.go:117] "RemoveContainer" containerID="823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e"
Jul 07 02:29:48.948471 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.948451    2566 scope.go:117] "RemoveContainer" containerID="f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd"
Jul 07 02:29:48.953951 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.953929    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"]
Jul 07 02:29:48.956354 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.956341    2566 scope.go:117] "RemoveContainer" containerID="4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1"
Jul 07 02:29:48.959783 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.959763    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-699589c4f4-xrdkf"]
Jul 07 02:29:48.963645 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.963625    2566 scope.go:117] "RemoveContainer" containerID="e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9"
Jul 07 02:29:48.963905 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:29:48.963887    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9\": container with ID starting with e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9 not found: ID does not exist" containerID="e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9"
Jul 07 02:29:48.963955 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.963914    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9"} err="failed to get container status \"e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9\": rpc error: code = NotFound desc = could not find container \"e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9\": container with ID starting with e11c21f34ea73dc0d4c03a44a5cef21309164f9871903a7afc8430347dbf1fa9 not found: ID does not exist"
Jul 07 02:29:48.963955 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.963932    2566 scope.go:117] "RemoveContainer" containerID="823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e"
Jul 07 02:29:48.964161 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:29:48.964147    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e\": container with ID starting with 823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e not found: ID does not exist" containerID="823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e"
Jul 07 02:29:48.964200 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.964165    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e"} err="failed to get container status \"823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e\": rpc error: code = NotFound desc = could not find container \"823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e\": container with ID starting with 823db6a1515b3496d0a2eea9114b3993988042b6969a82f5b582642882a05b7e not found: ID does not exist"
Jul 07 02:29:48.964200 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.964177    2566 scope.go:117] "RemoveContainer" containerID="f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd"
Jul 07 02:29:48.964399 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:29:48.964383    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd\": container with ID starting with f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd not found: ID does not exist" containerID="f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd"
Jul 07 02:29:48.964447 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.964401    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd"} err="failed to get container status \"f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd\": rpc error: code = NotFound desc = could not find container \"f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd\": container with ID starting with f98af9b734da81fdf727fbb8ba16e62755a9ac39737a56e86c96e27bc0d33bbd not found: ID does not exist"
Jul 07 02:29:48.964447 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.964416    2566 scope.go:117] "RemoveContainer" containerID="4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1"
Jul 07 02:29:48.964625 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:29:48.964606    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1\": container with ID starting with 4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1 not found: ID does not exist" containerID="4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1"
Jul 07 02:29:48.964663 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:48.964630    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1"} err="failed to get container status \"4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1\": rpc error: code = NotFound desc = could not find container \"4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1\": container with ID starting with 4865fa11beee7e7de5db4b87d22763de23d7bfad69514cbd9bfd7250bc2179d1 not found: ID does not exist"
Jul 07 02:29:50.829986 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:50.829949    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:29:50.830436 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:50.830299    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:29:50.848199 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:29:50.848170    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" path="/var/lib/kubelet/pods/298db630-4dd9-46b0-9bad-24d8347b5211/volumes"
Jul 07 02:30:00.830326 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:00.830277    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:30:00.830811 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:00.830707    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:30:10.830570 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:10.830516    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:30:10.831043 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:10.830996    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:30:20.830175 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:20.830133    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:30:20.830626 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:20.830520    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:30:30.830172 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:30.830121    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:30:30.830617 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:30.830469    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:30:40.830499 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:40.830465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:30:40.830921 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:40.830648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:30:53.313958 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.313925    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"]
Jul 07 02:30:53.314382 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.314181    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" containerID="cri-o://92b85a47c8c6d3347507543e3fb0cb4449006f85fe62c509ce9cd5a6daec15d7" gracePeriod=30
Jul 07 02:30:53.314382 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.314212    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" containerID="cri-o://444ad6992a4dcaae5888f04f58f7ab52e2901f2a4fb1a3d0e10a546e326dc2bc" gracePeriod=30
Jul 07 02:30:53.314382 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.314228    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" containerID="cri-o://182bb6312f9cb5401bb6d10435922674d74bba8a4b708648a5ae42a52d63de07" gracePeriod=30
Jul 07 02:30:53.438641 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.438603    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"]
Jul 07 02:30:53.439578 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.439550    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy"
Jul 07 02:30:53.439832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.439817    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy"
Jul 07 02:30:53.439939 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.439930    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent"
Jul 07 02:30:53.440006 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.439998    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent"
Jul 07 02:30:53.440087 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440078    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="storage-initializer"
Jul 07 02:30:53.440143 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440135    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="storage-initializer"
Jul 07 02:30:53.440233 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440224    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container"
Jul 07 02:30:53.440298 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440290    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container"
Jul 07 02:30:53.440525 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440511    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="agent"
Jul 07 02:30:53.440622 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440611    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kube-rbac-proxy"
Jul 07 02:30:53.440732 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.440721    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="298db630-4dd9-46b0-9bad-24d8347b5211" containerName="kserve-container"
Jul 07 02:30:53.445015 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.444992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.447369 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.447338    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 07 02:30:53.447476 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.447375    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 07 02:30:53.452146 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.452123    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"]
Jul 07 02:30:53.542390 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.542352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.542569 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.542397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfh2\" (UniqueName: \"kubernetes.io/projected/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-api-access-kkfh2\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.542569 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.542450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.643191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.643102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.643191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.643136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfh2\" (UniqueName: \"kubernetes.io/projected/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-api-access-kkfh2\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.643191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.643185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.643432 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:30:53.643359    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 07 02:30:53.643478 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:30:53.643448    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls podName:ec52aeea-e796-4f8c-9c95-33e2f7b71bf0 nodeName:}" failed. No retries permitted until 2026-07-07 02:30:54.143428111 +0000 UTC m=+789.802536857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-x5r4r" (UID: "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0") : secret "message-dumper-predictor-serving-cert" not found
Jul 07 02:30:53.643799 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.643778    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:53.657952 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:53.657927    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfh2\" (UniqueName: \"kubernetes.io/projected/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-api-access-kkfh2\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:54.146557 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.146519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:54.149016 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.148989    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-x5r4r\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:54.226954 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.226913    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerID="182bb6312f9cb5401bb6d10435922674d74bba8a4b708648a5ae42a52d63de07" exitCode=2
Jul 07 02:30:54.227112 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.226963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerDied","Data":"182bb6312f9cb5401bb6d10435922674d74bba8a4b708648a5ae42a52d63de07"}
Jul 07 02:30:54.358778 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.358740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:54.491176 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.491151    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"]
Jul 07 02:30:54.493077 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:30:54.493050    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec52aeea_e796_4f8c_9c95_33e2f7b71bf0.slice/crio-0f1d81ac721b308df1ed50de41a8a338fec1bc8752264e853c91c9c1f112ff62 WatchSource:0}: Error finding container 0f1d81ac721b308df1ed50de41a8a338fec1bc8752264e853c91c9c1f112ff62: Status 404 returned error can't find the container with id 0f1d81ac721b308df1ed50de41a8a338fec1bc8752264e853c91c9c1f112ff62
Jul 07 02:30:54.495034 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:54.495017    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:30:55.232551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:55.232513    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" event={"ID":"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0","Type":"ContainerStarted","Data":"0f1d81ac721b308df1ed50de41a8a338fec1bc8752264e853c91c9c1f112ff62"}
Jul 07 02:30:55.824719 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:55.824653    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 02:30:56.238896 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:56.238853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" event={"ID":"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0","Type":"ContainerStarted","Data":"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"}
Jul 07 02:30:56.239052 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:56.238926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" event={"ID":"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0","Type":"ContainerStarted","Data":"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"}
Jul 07 02:30:56.239052 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:56.239004    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:56.262357 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:56.262309    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" podStartSLOduration=1.912551581 podStartE2EDuration="3.262287931s" totalImagesPullingTime="1.34973635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:56.261808357 +0000 UTC m=+791.920917131" watchObservedRunningTime="2026-07-07 02:30:56.262287931 +0000 UTC m=+791.921396693"
Jul 07 02:30:57.243615 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:57.243580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:57.245108 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:57.245088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:30:58.249917 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:58.249886    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerID="92b85a47c8c6d3347507543e3fb0cb4449006f85fe62c509ce9cd5a6daec15d7" exitCode=0
Jul 07 02:30:58.250301 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:30:58.249960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerDied","Data":"92b85a47c8c6d3347507543e3fb0cb4449006f85fe62c509ce9cd5a6daec15d7"}
Jul 07 02:31:00.824870 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:00.824833    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 02:31:00.830289 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:00.830261    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:31:00.830604 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:00.830565    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:04.262715 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:04.262661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:31:05.824926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:05.824880    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 02:31:05.825367 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:05.825052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:31:10.824704 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:10.824589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 02:31:10.829985 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:10.829948    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:31:10.830310 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:10.830282    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:13.474235 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.474188    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"]
Jul 07 02:31:13.478170 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.478145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.480861 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.480835    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 07 02:31:13.480967 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.480845    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 07 02:31:13.495398 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.495301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"]
Jul 07 02:31:13.506486 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.506459    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.506630 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.506513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.506707 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.506633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfgs\" (UniqueName: \"kubernetes.io/projected/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-api-access-6lfgs\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.506755 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.506724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-cabundle-cert\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.506812 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.506794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kserve-provision-location\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.607700 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.607642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.607888 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.607727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.607888 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.607756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfgs\" (UniqueName: \"kubernetes.io/projected/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-api-access-6lfgs\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.607888 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.607799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-cabundle-cert\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.608051 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:31:13.607884    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 07 02:31:13.608051 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.607907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kserve-provision-location\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.608051 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:31:13.607948    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls podName:77a049e2-edd7-4b44-9481-0e8f9d1dcda6 nodeName:}" failed. No retries permitted until 2026-07-07 02:31:14.107927629 +0000 UTC m=+809.767036378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls") pod "isvc-logger-predictor-56cd5d9d45-p4whk" (UID: "77a049e2-edd7-4b44-9481-0e8f9d1dcda6") : secret "isvc-logger-predictor-serving-cert" not found
Jul 07 02:31:13.608272 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.608253    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kserve-provision-location\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.608458 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.608439    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.608515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.608461    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-cabundle-cert\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:13.617168 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:13.617139    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfgs\" (UniqueName: \"kubernetes.io/projected/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-api-access-6lfgs\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:14.112778 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:14.112737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:14.114905 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:14.114887    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls\") pod \"isvc-logger-predictor-56cd5d9d45-p4whk\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:14.388549 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:14.388455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:14.541134 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:14.541102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"]
Jul 07 02:31:14.544965 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:31:14.544940    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a049e2_edd7_4b44_9481_0e8f9d1dcda6.slice/crio-01a89ff70fe690ea70cefe87dd9496948bb54d623666ccf24fa2d0a291c10afd WatchSource:0}: Error finding container 01a89ff70fe690ea70cefe87dd9496948bb54d623666ccf24fa2d0a291c10afd: Status 404 returned error can't find the container with id 01a89ff70fe690ea70cefe87dd9496948bb54d623666ccf24fa2d0a291c10afd
Jul 07 02:31:15.328958 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:15.328916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerStarted","Data":"ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f"}
Jul 07 02:31:15.329126 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:15.328982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerStarted","Data":"01a89ff70fe690ea70cefe87dd9496948bb54d623666ccf24fa2d0a291c10afd"}
Jul 07 02:31:15.824952 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:15.824903    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 02:31:18.344186 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:18.344147    2566 generic.go:356] "Generic (PLEG): container finished" podID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerID="ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f" exitCode=0
Jul 07 02:31:18.344560 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:18.344221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerDied","Data":"ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f"}
Jul 07 02:31:19.351104 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.351069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerStarted","Data":"e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44"}
Jul 07 02:31:19.351477 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.351136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerStarted","Data":"7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3"}
Jul 07 02:31:19.351477 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.351160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerStarted","Data":"d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff"}
Jul 07 02:31:19.351477 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.351432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:19.351603 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.351574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:19.352999 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.352973    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:19.375982 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:19.375938    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podStartSLOduration=2.843314003 podStartE2EDuration="6.375922663s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.53260866s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:19.374251255 +0000 UTC m=+815.033360020" watchObservedRunningTime="2026-07-07 02:31:19.375922663 +0000 UTC m=+815.035031426"
Jul 07 02:31:20.355125 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.355090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:20.355515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.355243    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:20.356424 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.356386    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:20.824622 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.824584    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 07 02:31:20.830112 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.830080    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:5000: connect: connection refused"
Jul 07 02:31:20.830232 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.830207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:31:20.830435 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.830415    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:20.830510 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:20.830500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:31:21.359928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:21.359882    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:21.360356 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:21.360267    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:23.371996 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.371963    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerID="444ad6992a4dcaae5888f04f58f7ab52e2901f2a4fb1a3d0e10a546e326dc2bc" exitCode=0
Jul 07 02:31:23.372343 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.372034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerDied","Data":"444ad6992a4dcaae5888f04f58f7ab52e2901f2a4fb1a3d0e10a546e326dc2bc"}
Jul 07 02:31:23.471619 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.471596    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:31:23.594213 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bt8b\" (UniqueName: \"kubernetes.io/projected/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-api-access-8bt8b\") pod \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") "
Jul 07 02:31:23.594213 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-proxy-tls\") pod \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") "
Jul 07 02:31:23.594446 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-rbac-proxy-sar-config\") pod \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") "
Jul 07 02:31:23.594446 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594253    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kserve-provision-location\") pod \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") "
Jul 07 02:31:23.594446 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594296    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-cabundle-cert\") pod \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\" (UID: \"9ee872aa-f474-45fe-9f2a-7d0c937e99cb\") "
Jul 07 02:31:23.594632 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594607    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ee872aa-f474-45fe-9f2a-7d0c937e99cb" (UID: "9ee872aa-f474-45fe-9f2a-7d0c937e99cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:31:23.594713 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594642    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ee872aa-f474-45fe-9f2a-7d0c937e99cb" (UID: "9ee872aa-f474-45fe-9f2a-7d0c937e99cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:31:23.594713 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.594656    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ee872aa-f474-45fe-9f2a-7d0c937e99cb" (UID: "9ee872aa-f474-45fe-9f2a-7d0c937e99cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:31:23.596371 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.596349    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ee872aa-f474-45fe-9f2a-7d0c937e99cb" (UID: "9ee872aa-f474-45fe-9f2a-7d0c937e99cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:31:23.596451 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.596408    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-api-access-8bt8b" (OuterVolumeSpecName: "kube-api-access-8bt8b") pod "9ee872aa-f474-45fe-9f2a-7d0c937e99cb" (UID: "9ee872aa-f474-45fe-9f2a-7d0c937e99cb"). InnerVolumeSpecName "kube-api-access-8bt8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:31:23.694991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.694954    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bt8b\" (UniqueName: \"kubernetes.io/projected/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-api-access-8bt8b\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:31:23.694991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.694983    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:31:23.694991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.694994    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:31:23.694991 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.695002    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:31:23.695229 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:23.695011    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ee872aa-f474-45fe-9f2a-7d0c937e99cb-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:31:24.379030 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.378994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb" event={"ID":"9ee872aa-f474-45fe-9f2a-7d0c937e99cb","Type":"ContainerDied","Data":"51dd73af9bf9dd9705f4c187f4c7f097f95178fc04d27c1aa39e141d1562ffa4"}
Jul 07 02:31:24.379030 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.379022    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"
Jul 07 02:31:24.379487 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.379075    2566 scope.go:117] "RemoveContainer" containerID="444ad6992a4dcaae5888f04f58f7ab52e2901f2a4fb1a3d0e10a546e326dc2bc"
Jul 07 02:31:24.388312 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.388290    2566 scope.go:117] "RemoveContainer" containerID="182bb6312f9cb5401bb6d10435922674d74bba8a4b708648a5ae42a52d63de07"
Jul 07 02:31:24.395635 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.395615    2566 scope.go:117] "RemoveContainer" containerID="92b85a47c8c6d3347507543e3fb0cb4449006f85fe62c509ce9cd5a6daec15d7"
Jul 07 02:31:24.403187 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.403159    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"]
Jul 07 02:31:24.403892 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.403851    2566 scope.go:117] "RemoveContainer" containerID="38d20df642cd7199dc5a4f57d66b18eb6274390f38b72bbd8a31dbff7fefe997"
Jul 07 02:31:24.406615 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.406592    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7b84bc55cc-p86qb"]
Jul 07 02:31:24.848056 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:24.848022    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" path="/var/lib/kubelet/pods/9ee872aa-f474-45fe-9f2a-7d0c937e99cb/volumes"
Jul 07 02:31:26.363805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:26.363777    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:31:26.364350 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:26.364312    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:26.364835 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:26.364803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:36.364398 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:36.364349    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:36.364952 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:36.364923    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:46.364380 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:46.364336    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:46.364875 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:46.364515    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:31:56.364411 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:56.364364    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:31:56.364935 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:31:56.364656    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:32:06.364762 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:06.364711    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:32:06.365299 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:06.365211    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:32:16.364778 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:16.364731    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:32:16.365253 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:16.365176    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:32:26.364493 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:26.364446    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:32:26.365080 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:26.365019    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:32:36.364820 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:36.364783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:32:36.365392 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:36.365032    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:32:44.770344 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:44.770266    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:32:44.771782 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:44.771759    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:32:44.778425 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:44.778398    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:32:44.779624 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:44.779605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:32:48.463418 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.463392    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-x5r4r_ec52aeea-e796-4f8c-9c95-33e2f7b71bf0/kserve-container/0.log"
Jul 07 02:32:48.683299 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.683258    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"]
Jul 07 02:32:48.683595 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.683562    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" containerID="cri-o://d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff" gracePeriod=30
Jul 07 02:32:48.683733 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.683592    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" containerID="cri-o://e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44" gracePeriod=30
Jul 07 02:32:48.683805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.683606    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" containerID="cri-o://7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3" gracePeriod=30
Jul 07 02:32:48.770165 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770090    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"]
Jul 07 02:32:48.770629 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770608    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="storage-initializer"
Jul 07 02:32:48.770629 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770627    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="storage-initializer"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770657    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770666    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770696    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770705    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770716    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770724    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent"
Jul 07 02:32:48.770803 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770805    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kube-rbac-proxy"
Jul 07 02:32:48.771014 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770818    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="kserve-container"
Jul 07 02:32:48.771014 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.770829    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee872aa-f474-45fe-9f2a-7d0c937e99cb" containerName="agent"
Jul 07 02:32:48.775080 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.774942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.777341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.777315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 07 02:32:48.777341 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.777332    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 07 02:32:48.777699 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.777657    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"]
Jul 07 02:32:48.777913 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.777877    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kserve-container" containerID="cri-o://cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1" gracePeriod=30
Jul 07 02:32:48.777974 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.777951    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kube-rbac-proxy" containerID="cri-o://47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263" gracePeriod=30
Jul 07 02:32:48.786750 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.786727    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"]
Jul 07 02:32:48.812051 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.812025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2532d562-59a7-479d-a104-b960d7e0be8e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.812180 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.812055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.812180 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.812077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.812289 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.812175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbt27\" (UniqueName: \"kubernetes.io/projected/2532d562-59a7-479d-a104-b960d7e0be8e-kube-api-access-hbt27\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.812289 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.812204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.912611 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.912580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbt27\" (UniqueName: \"kubernetes.io/projected/2532d562-59a7-479d-a104-b960d7e0be8e-kube-api-access-hbt27\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.912611 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.912619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.912868 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.912641    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2532d562-59a7-479d-a104-b960d7e0be8e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.912868 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.912732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.912868 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.912783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.913037 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.912979    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2532d562-59a7-479d-a104-b960d7e0be8e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.913037 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:32:48.913019    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 07 02:32:48.913137 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:32:48.913075    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls podName:2532d562-59a7-479d-a104-b960d7e0be8e nodeName:}" failed. No retries permitted until 2026-07-07 02:32:49.413056022 +0000 UTC m=+905.072164767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" (UID: "2532d562-59a7-479d-a104-b960d7e0be8e") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 07 02:32:48.913536 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.913511    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.913625 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.913512    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:48.924938 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:48.923378    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbt27\" (UniqueName: \"kubernetes.io/projected/2532d562-59a7-479d-a104-b960d7e0be8e-kube-api-access-hbt27\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:49.032631 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.032608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:32:49.114758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.114729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-rbac-proxy-sar-config\") pod \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") "
Jul 07 02:32:49.114924 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.114795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkfh2\" (UniqueName: \"kubernetes.io/projected/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-api-access-kkfh2\") pod \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") "
Jul 07 02:32:49.115170 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.115144    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" (UID: "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:32:49.117222 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.117203    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-api-access-kkfh2" (OuterVolumeSpecName: "kube-api-access-kkfh2") pod "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" (UID: "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0"). InnerVolumeSpecName "kube-api-access-kkfh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:32:49.215183 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.215144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls\") pod \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\" (UID: \"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0\") "
Jul 07 02:32:49.215332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.215297    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfh2\" (UniqueName: \"kubernetes.io/projected/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-api-access-kkfh2\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:32:49.215332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.215308    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:32:49.217197 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.217165    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" (UID: "ec52aeea-e796-4f8c-9c95-33e2f7b71bf0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:32:49.316029 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.315949    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:32:49.416542 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.416502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:49.418851 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.418826    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:49.469739 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.469715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:32:49.601119 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.601093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"]
Jul 07 02:32:49.603441 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:32:49.603418    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2532d562_59a7_479d_a104_b960d7e0be8e.slice/crio-5baadb017f1fc1515d6bb6f9f06fbe4eff803b6ba8224111104a3db3147fc2ef WatchSource:0}: Error finding container 5baadb017f1fc1515d6bb6f9f06fbe4eff803b6ba8224111104a3db3147fc2ef: Status 404 returned error can't find the container with id 5baadb017f1fc1515d6bb6f9f06fbe4eff803b6ba8224111104a3db3147fc2ef
Jul 07 02:32:49.774898 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.774868    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerID="47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263" exitCode=2
Jul 07 02:32:49.774898 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.774893    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerID="cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1" exitCode=2
Jul 07 02:32:49.775117 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.774926    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"
Jul 07 02:32:49.775117 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.774966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" event={"ID":"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0","Type":"ContainerDied","Data":"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"}
Jul 07 02:32:49.775117 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.775041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" event={"ID":"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0","Type":"ContainerDied","Data":"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"}
Jul 07 02:32:49.775117 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.775077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r" event={"ID":"ec52aeea-e796-4f8c-9c95-33e2f7b71bf0","Type":"ContainerDied","Data":"0f1d81ac721b308df1ed50de41a8a338fec1bc8752264e853c91c9c1f112ff62"}
Jul 07 02:32:49.775305 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.775119    2566 scope.go:117] "RemoveContainer" containerID="47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"
Jul 07 02:32:49.776779 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.776735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerStarted","Data":"d0b9f9bd1837352af52e8b0ee655c7ceca0b2b2226a14045bab52a6ec7be7593"}
Jul 07 02:32:49.776920 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.776807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerStarted","Data":"5baadb017f1fc1515d6bb6f9f06fbe4eff803b6ba8224111104a3db3147fc2ef"}
Jul 07 02:32:49.779767 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.779747    2566 generic.go:356] "Generic (PLEG): container finished" podID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerID="7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3" exitCode=2
Jul 07 02:32:49.779910 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.779811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerDied","Data":"7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3"}
Jul 07 02:32:49.784942 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.784924    2566 scope.go:117] "RemoveContainer" containerID="cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"
Jul 07 02:32:49.792729 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.792709    2566 scope.go:117] "RemoveContainer" containerID="47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"
Jul 07 02:32:49.792985 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:32:49.792964    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263\": container with ID starting with 47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263 not found: ID does not exist" containerID="47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"
Jul 07 02:32:49.793073 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.792990    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"} err="failed to get container status \"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263\": rpc error: code = NotFound desc = could not find container \"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263\": container with ID starting with 47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263 not found: ID does not exist"
Jul 07 02:32:49.793073 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.793008    2566 scope.go:117] "RemoveContainer" containerID="cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"
Jul 07 02:32:49.793248 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:32:49.793226    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1\": container with ID starting with cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1 not found: ID does not exist" containerID="cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"
Jul 07 02:32:49.793315 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.793253    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"} err="failed to get container status \"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1\": rpc error: code = NotFound desc = could not find container \"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1\": container with ID starting with cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1 not found: ID does not exist"
Jul 07 02:32:49.793315 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.793270    2566 scope.go:117] "RemoveContainer" containerID="47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"
Jul 07 02:32:49.793459 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.793440    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263"} err="failed to get container status \"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263\": rpc error: code = NotFound desc = could not find container \"47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263\": container with ID starting with 47ed5dc32e5cdc683efe591d3be576bab97368784ea1ef8e403be09f059ff263 not found: ID does not exist"
Jul 07 02:32:49.793459 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.793457    2566 scope.go:117] "RemoveContainer" containerID="cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"
Jul 07 02:32:49.793631 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.793614    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1"} err="failed to get container status \"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1\": rpc error: code = NotFound desc = could not find container \"cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1\": container with ID starting with cfaf0eaaa06880598a8cd7e9387d60f11e8291c010a3fec2f3a18deef47400f1 not found: ID does not exist"
Jul 07 02:32:49.811267 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.811235    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"]
Jul 07 02:32:49.815618 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:49.815582    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-x5r4r"]
Jul 07 02:32:50.847799 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:50.847766    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" path="/var/lib/kubelet/pods/ec52aeea-e796-4f8c-9c95-33e2f7b71bf0/volumes"
Jul 07 02:32:51.360707 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:51.360651    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 07 02:32:53.801834 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:53.801797    2566 generic.go:356] "Generic (PLEG): container finished" podID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerID="d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff" exitCode=0
Jul 07 02:32:53.802208 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:53.801875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerDied","Data":"d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff"}
Jul 07 02:32:56.360264 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:56.360223    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 07 02:32:56.364895 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:56.364864    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:32:56.365200 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:32:56.365176    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:33:01.360146 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:01.360099    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 07 02:33:01.360562 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:01.360261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:33:06.359974 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:06.359934    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 07 02:33:06.364294 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:06.364253    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:33:06.364580 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:06.364562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:33:11.360416 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:11.360370    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 07 02:33:16.360842 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:16.360797    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 07 02:33:16.365153 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:16.365129    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 07 02:33:16.365278 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:16.365264    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:33:16.365540 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:16.365500    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:33:16.365633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:16.365622    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:33:18.839908 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.839880    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:33:18.911030 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.910993    2566 generic.go:356] "Generic (PLEG): container finished" podID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerID="e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44" exitCode=0
Jul 07 02:33:18.911194 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.911061    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"
Jul 07 02:33:18.911194 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.911071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerDied","Data":"e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44"}
Jul 07 02:33:18.911194 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.911140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk" event={"ID":"77a049e2-edd7-4b44-9481-0e8f9d1dcda6","Type":"ContainerDied","Data":"01a89ff70fe690ea70cefe87dd9496948bb54d623666ccf24fa2d0a291c10afd"}
Jul 07 02:33:18.911194 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.911171    2566 scope.go:117] "RemoveContainer" containerID="e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44"
Jul 07 02:33:18.918992 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.918973    2566 scope.go:117] "RemoveContainer" containerID="7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3"
Jul 07 02:33:18.926317 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.926298    2566 scope.go:117] "RemoveContainer" containerID="d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff"
Jul 07 02:33:18.933616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.933600    2566 scope.go:117] "RemoveContainer" containerID="ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f"
Jul 07 02:33:18.940684 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.940658    2566 scope.go:117] "RemoveContainer" containerID="e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44"
Jul 07 02:33:18.940926 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:33:18.940906    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44\": container with ID starting with e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44 not found: ID does not exist" containerID="e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44"
Jul 07 02:33:18.940972 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.940935    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44"} err="failed to get container status \"e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44\": rpc error: code = NotFound desc = could not find container \"e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44\": container with ID starting with e12ba55a38c91aeec62fe7a6a71b6094c5a343e38449de6c1ec93d45a4b58e44 not found: ID does not exist"
Jul 07 02:33:18.940972 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.940953    2566 scope.go:117] "RemoveContainer" containerID="7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3"
Jul 07 02:33:18.941200 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:33:18.941182    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3\": container with ID starting with 7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3 not found: ID does not exist" containerID="7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3"
Jul 07 02:33:18.941242 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.941207    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3"} err="failed to get container status \"7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3\": rpc error: code = NotFound desc = could not find container \"7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3\": container with ID starting with 7848aec8337bf6be49f5c665af4e9e3c74d7b13ac24ef7c8bd22313e9a79a8e3 not found: ID does not exist"
Jul 07 02:33:18.941242 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.941222    2566 scope.go:117] "RemoveContainer" containerID="d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff"
Jul 07 02:33:18.941498 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:33:18.941480    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff\": container with ID starting with d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff not found: ID does not exist" containerID="d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff"
Jul 07 02:33:18.941551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.941502    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff"} err="failed to get container status \"d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff\": rpc error: code = NotFound desc = could not find container \"d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff\": container with ID starting with d97b96e6b10168d3befd4252bdf90fc985c2632a210b832a7740b8980609f2ff not found: ID does not exist"
Jul 07 02:33:18.941551 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.941516    2566 scope.go:117] "RemoveContainer" containerID="ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f"
Jul 07 02:33:18.941746 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:33:18.941728    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f\": container with ID starting with ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f not found: ID does not exist" containerID="ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f"
Jul 07 02:33:18.941746 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.941746    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f"} err="failed to get container status \"ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f\": rpc error: code = NotFound desc = could not find container \"ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f\": container with ID starting with ce9c999dc650e67399ee7b43bb3d5971bb9bda33397126bb291d350fc248cb6f not found: ID does not exist"
Jul 07 02:33:18.971060 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kserve-provision-location\") pod \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") "
Jul 07 02:33:18.971210 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-rbac-proxy-sar-config\") pod \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") "
Jul 07 02:33:18.971210 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971095    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls\") pod \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") "
Jul 07 02:33:18.971210 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971129    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-cabundle-cert\") pod \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") "
Jul 07 02:33:18.971210 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971180    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfgs\" (UniqueName: \"kubernetes.io/projected/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-api-access-6lfgs\") pod \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\" (UID: \"77a049e2-edd7-4b44-9481-0e8f9d1dcda6\") "
Jul 07 02:33:18.971443 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971410    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77a049e2-edd7-4b44-9481-0e8f9d1dcda6" (UID: "77a049e2-edd7-4b44-9481-0e8f9d1dcda6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:33:18.971506 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971431    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77a049e2-edd7-4b44-9481-0e8f9d1dcda6" (UID: "77a049e2-edd7-4b44-9481-0e8f9d1dcda6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:33:18.971506 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971495    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77a049e2-edd7-4b44-9481-0e8f9d1dcda6" (UID: "77a049e2-edd7-4b44-9481-0e8f9d1dcda6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:33:18.971642 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.971619    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:33:18.973248 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.973225    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77a049e2-edd7-4b44-9481-0e8f9d1dcda6" (UID: "77a049e2-edd7-4b44-9481-0e8f9d1dcda6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:33:18.973360 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:18.973296    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-api-access-6lfgs" (OuterVolumeSpecName: "kube-api-access-6lfgs") pod "77a049e2-edd7-4b44-9481-0e8f9d1dcda6" (UID: "77a049e2-edd7-4b44-9481-0e8f9d1dcda6"). InnerVolumeSpecName "kube-api-access-6lfgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:33:19.072872 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.072842    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:33:19.072872 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.072871    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:33:19.072872 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.072882    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:33:19.073087 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.072890    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfgs\" (UniqueName: \"kubernetes.io/projected/77a049e2-edd7-4b44-9481-0e8f9d1dcda6-kube-api-access-6lfgs\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:33:19.234124 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.234089    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"]
Jul 07 02:33:19.240946 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.240913    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-56cd5d9d45-p4whk"]
Jul 07 02:33:19.917404 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.917370    2566 generic.go:356] "Generic (PLEG): container finished" podID="2532d562-59a7-479d-a104-b960d7e0be8e" containerID="d0b9f9bd1837352af52e8b0ee655c7ceca0b2b2226a14045bab52a6ec7be7593" exitCode=0
Jul 07 02:33:19.917816 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:19.917445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerDied","Data":"d0b9f9bd1837352af52e8b0ee655c7ceca0b2b2226a14045bab52a6ec7be7593"}
Jul 07 02:33:20.855886 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:33:20.855841    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" path="/var/lib/kubelet/pods/77a049e2-edd7-4b44-9481-0e8f9d1dcda6/volumes"
Jul 07 02:35:37.606172 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:37.606134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerStarted","Data":"02b4833470ee0138ba4cb3e5b40f7b24c2dc70e4f5dd6be72f7a9271a3f56542"}
Jul 07 02:35:37.606644 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:37.606192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerStarted","Data":"a1f545547107d48d1f186c4ca292c2dc5ef4d1db2f1156f33e33814795084633"}
Jul 07 02:35:37.606644 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:37.606216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:35:37.606644 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:37.606242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:35:37.607749 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:37.607716    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 02:35:37.639832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:37.639781    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" podStartSLOduration=3.258493452 podStartE2EDuration="2m49.639765412s" totalImagesPullingTime="2m16.772025084s" totalInitContainerRuntime="29.609246876s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:35:37.638275831 +0000 UTC m=+1073.297384606" watchObservedRunningTime="2026-07-07 02:35:37.639765412 +0000 UTC m=+1073.298874174"
Jul 07 02:35:38.610881 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:38.610839    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 02:35:43.616994 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:43.616961    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:35:43.618167 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:43.618134    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 07 02:35:53.619727 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:53.619694    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:35:58.909943 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:58.909901    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"]
Jul 07 02:35:58.910344 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:58.910186    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container" containerID="cri-o://a1f545547107d48d1f186c4ca292c2dc5ef4d1db2f1156f33e33814795084633" gracePeriod=30
Jul 07 02:35:58.910344 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:58.910272    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kube-rbac-proxy" containerID="cri-o://02b4833470ee0138ba4cb3e5b40f7b24c2dc70e4f5dd6be72f7a9271a3f56542" gracePeriod=30
Jul 07 02:35:59.703665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.703626    2566 generic.go:356] "Generic (PLEG): container finished" podID="2532d562-59a7-479d-a104-b960d7e0be8e" containerID="02b4833470ee0138ba4cb3e5b40f7b24c2dc70e4f5dd6be72f7a9271a3f56542" exitCode=2
Jul 07 02:35:59.703665 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.703659    2566 generic.go:356] "Generic (PLEG): container finished" podID="2532d562-59a7-479d-a104-b960d7e0be8e" containerID="a1f545547107d48d1f186c4ca292c2dc5ef4d1db2f1156f33e33814795084633" exitCode=0
Jul 07 02:35:59.703896 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.703700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerDied","Data":"02b4833470ee0138ba4cb3e5b40f7b24c2dc70e4f5dd6be72f7a9271a3f56542"}
Jul 07 02:35:59.703896 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.703755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerDied","Data":"a1f545547107d48d1f186c4ca292c2dc5ef4d1db2f1156f33e33814795084633"}
Jul 07 02:35:59.807954 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.807922    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:35:59.918886 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.918797    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls\") pod \"2532d562-59a7-479d-a104-b960d7e0be8e\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") "
Jul 07 02:35:59.918886 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.918835    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-kube-rbac-proxy-sar-config\") pod \"2532d562-59a7-479d-a104-b960d7e0be8e\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") "
Jul 07 02:35:59.918886 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.918865    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2532d562-59a7-479d-a104-b960d7e0be8e-kserve-provision-location\") pod \"2532d562-59a7-479d-a104-b960d7e0be8e\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") "
Jul 07 02:35:59.919417 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.918898    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-cabundle-cert\") pod \"2532d562-59a7-479d-a104-b960d7e0be8e\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") "
Jul 07 02:35:59.919417 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.918931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbt27\" (UniqueName: \"kubernetes.io/projected/2532d562-59a7-479d-a104-b960d7e0be8e-kube-api-access-hbt27\") pod \"2532d562-59a7-479d-a104-b960d7e0be8e\" (UID: \"2532d562-59a7-479d-a104-b960d7e0be8e\") "
Jul 07 02:35:59.919417 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.919250    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2532d562-59a7-479d-a104-b960d7e0be8e" (UID: "2532d562-59a7-479d-a104-b960d7e0be8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:35:59.919417 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.919249    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2532d562-59a7-479d-a104-b960d7e0be8e" (UID: "2532d562-59a7-479d-a104-b960d7e0be8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:35:59.920860 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.920840    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2532d562-59a7-479d-a104-b960d7e0be8e" (UID: "2532d562-59a7-479d-a104-b960d7e0be8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:35:59.921156 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.921130    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2532d562-59a7-479d-a104-b960d7e0be8e-kube-api-access-hbt27" (OuterVolumeSpecName: "kube-api-access-hbt27") pod "2532d562-59a7-479d-a104-b960d7e0be8e" (UID: "2532d562-59a7-479d-a104-b960d7e0be8e"). InnerVolumeSpecName "kube-api-access-hbt27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:35:59.923108 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:35:59.923084    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2532d562-59a7-479d-a104-b960d7e0be8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2532d562-59a7-479d-a104-b960d7e0be8e" (UID: "2532d562-59a7-479d-a104-b960d7e0be8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:00.019806 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.019777    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2532d562-59a7-479d-a104-b960d7e0be8e-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:36:00.019806 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.019809    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:36:00.020035 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.019825    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2532d562-59a7-479d-a104-b960d7e0be8e-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:36:00.020035 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.019841    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2532d562-59a7-479d-a104-b960d7e0be8e-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:36:00.020035 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.019855    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbt27\" (UniqueName: \"kubernetes.io/projected/2532d562-59a7-479d-a104-b960d7e0be8e-kube-api-access-hbt27\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:36:00.709620 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.709591    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"
Jul 07 02:36:00.709620 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.709612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc" event={"ID":"2532d562-59a7-479d-a104-b960d7e0be8e","Type":"ContainerDied","Data":"5baadb017f1fc1515d6bb6f9f06fbe4eff803b6ba8224111104a3db3147fc2ef"}
Jul 07 02:36:00.709873 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.709698    2566 scope.go:117] "RemoveContainer" containerID="02b4833470ee0138ba4cb3e5b40f7b24c2dc70e4f5dd6be72f7a9271a3f56542"
Jul 07 02:36:00.718717 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.718696    2566 scope.go:117] "RemoveContainer" containerID="a1f545547107d48d1f186c4ca292c2dc5ef4d1db2f1156f33e33814795084633"
Jul 07 02:36:00.726012 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.725997    2566 scope.go:117] "RemoveContainer" containerID="d0b9f9bd1837352af52e8b0ee655c7ceca0b2b2226a14045bab52a6ec7be7593"
Jul 07 02:36:00.733292 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.733220    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"]
Jul 07 02:36:00.735580 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.735561    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-7vlmc"]
Jul 07 02:36:00.848279 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:00.848245    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" path="/var/lib/kubelet/pods/2532d562-59a7-479d-a104-b960d7e0be8e/volumes"
Jul 07 02:36:04.260865 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.260815    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"]
Jul 07 02:36:04.261332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261260    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261278    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261290    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261298    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261314    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent"
Jul 07 02:36:04.261332 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261322    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261339    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261347    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261358    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="storage-initializer"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261366    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="storage-initializer"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261382    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="storage-initializer"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261391    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="storage-initializer"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261412    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261420    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261430    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261441    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261452    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261461    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261534    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261546    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261557    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="agent"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261565    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261576    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a049e2-edd7-4b44-9481-0e8f9d1dcda6" containerName="kube-rbac-proxy"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261583    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2532d562-59a7-479d-a104-b960d7e0be8e" containerName="kserve-container"
Jul 07 02:36:04.261633 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.261593    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec52aeea-e796-4f8c-9c95-33e2f7b71bf0" containerName="kube-rbac-proxy"
Jul 07 02:36:04.284685 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.284638    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"]
Jul 07 02:36:04.284869 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.284803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.287747 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.287538    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 07 02:36:04.287900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.287745    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:36:04.287900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.287614    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:36:04.287900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.287798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:36:04.287900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.287604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:36:04.287900 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.287746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 07 02:36:04.288523 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.288503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76zzm\""
Jul 07 02:36:04.355771 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.355737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.355935 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.355785    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.355935 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.355900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdt76\" (UniqueName: \"kubernetes.io/projected/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-api-access-xdt76\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.356020 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.355948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.356020 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.356010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.457501 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.457450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.457743 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.457514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.457743 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.457554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdt76\" (UniqueName: \"kubernetes.io/projected/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-api-access-xdt76\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.457743 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.457607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.457743 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:36:04.457622    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 07 02:36:04.457743 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.457664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.457743 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:36:04.457715    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls podName:a2bcd02a-a85c-4f08-aa68-b6bf85e02933 nodeName:}" failed. No retries permitted until 2026-07-07 02:36:04.957692739 +0000 UTC m=+1100.616801482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-p7ntg" (UID: "a2bcd02a-a85c-4f08-aa68-b6bf85e02933") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 07 02:36:04.458094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.458071    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.458308 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.458288    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.458373 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.458287    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.469237 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.469202    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdt76\" (UniqueName: \"kubernetes.io/projected/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-api-access-xdt76\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.961808 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.961762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:04.964394 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:04.964368    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-p7ntg\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:05.196742 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:05.196694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:05.356625 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:05.356595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"]
Jul 07 02:36:05.359126 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:36:05.359098    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bcd02a_a85c_4f08_aa68_b6bf85e02933.slice/crio-dc2172b975ebe117b836812958cbc163f702899d622ddb13875079aab6c2816d WatchSource:0}: Error finding container dc2172b975ebe117b836812958cbc163f702899d622ddb13875079aab6c2816d: Status 404 returned error can't find the container with id dc2172b975ebe117b836812958cbc163f702899d622ddb13875079aab6c2816d
Jul 07 02:36:05.737684 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:05.737646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerStarted","Data":"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"}
Jul 07 02:36:05.737843 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:05.737726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerStarted","Data":"dc2172b975ebe117b836812958cbc163f702899d622ddb13875079aab6c2816d"}
Jul 07 02:36:35.867046 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:35.867006    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerID="452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec" exitCode=0
Jul 07 02:36:35.867509 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:35.867079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerDied","Data":"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"}
Jul 07 02:36:36.874439 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:36.874404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerStarted","Data":"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"}
Jul 07 02:36:36.874847 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:36.874470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerStarted","Data":"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"}
Jul 07 02:36:36.874847 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:36.874704    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:36.896816 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:36.896768    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" podStartSLOduration=3.276685137 podStartE2EDuration="32.896752336s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.620067199s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:36:36.894398991 +0000 UTC m=+1132.553507763" watchObservedRunningTime="2026-07-07 02:36:36.896752336 +0000 UTC m=+1132.555861096"
Jul 07 02:36:37.878988 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:37.878959    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:36:43.888051 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:36:43.888022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:37:44.796442 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:37:44.796413    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:37:44.798389 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:37:44.798363    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:37:44.803398 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:37:44.803375    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:37:44.805348 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:37:44.805327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:38:11.847338 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:11.847305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:38:14.421792 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:14.421757    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"]
Jul 07 02:38:14.422192 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:14.422039    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kserve-container" containerID="cri-o://ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee" gracePeriod=30
Jul 07 02:38:14.422192 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:14.422102    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kube-rbac-proxy" containerID="cri-o://d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4" gracePeriod=30
Jul 07 02:38:15.266958 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.266933    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:38:15.314883 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.314795    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerID="d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4" exitCode=2
Jul 07 02:38:15.314883 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.314824    2566 generic.go:356] "Generic (PLEG): container finished" podID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerID="ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee" exitCode=0
Jul 07 02:38:15.314883 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.314857    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"
Jul 07 02:38:15.315128 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.314876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerDied","Data":"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"}
Jul 07 02:38:15.315128 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.314986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerDied","Data":"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"}
Jul 07 02:38:15.315128 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.315020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg" event={"ID":"a2bcd02a-a85c-4f08-aa68-b6bf85e02933","Type":"ContainerDied","Data":"dc2172b975ebe117b836812958cbc163f702899d622ddb13875079aab6c2816d"}
Jul 07 02:38:15.315128 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.315057    2566 scope.go:117] "RemoveContainer" containerID="d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"
Jul 07 02:38:15.323709 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.323684    2566 scope.go:117] "RemoveContainer" containerID="ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"
Jul 07 02:38:15.331241 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.331220    2566 scope.go:117] "RemoveContainer" containerID="452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"
Jul 07 02:38:15.338531 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.338512    2566 scope.go:117] "RemoveContainer" containerID="d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"
Jul 07 02:38:15.338787 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:38:15.338764    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4\": container with ID starting with d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4 not found: ID does not exist" containerID="d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"
Jul 07 02:38:15.338871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.338800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"} err="failed to get container status \"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4\": rpc error: code = NotFound desc = could not find container \"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4\": container with ID starting with d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4 not found: ID does not exist"
Jul 07 02:38:15.338871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.338827    2566 scope.go:117] "RemoveContainer" containerID="ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"
Jul 07 02:38:15.339083 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:38:15.339063    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee\": container with ID starting with ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee not found: ID does not exist" containerID="ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"
Jul 07 02:38:15.339127 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339089    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"} err="failed to get container status \"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee\": rpc error: code = NotFound desc = could not find container \"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee\": container with ID starting with ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee not found: ID does not exist"
Jul 07 02:38:15.339127 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339105    2566 scope.go:117] "RemoveContainer" containerID="452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"
Jul 07 02:38:15.339323 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:38:15.339305    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec\": container with ID starting with 452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec not found: ID does not exist" containerID="452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"
Jul 07 02:38:15.339365 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339331    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"} err="failed to get container status \"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec\": rpc error: code = NotFound desc = could not find container \"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec\": container with ID starting with 452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec not found: ID does not exist"
Jul 07 02:38:15.339365 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339347    2566 scope.go:117] "RemoveContainer" containerID="d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"
Jul 07 02:38:15.339583 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339558    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4"} err="failed to get container status \"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4\": rpc error: code = NotFound desc = could not find container \"d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4\": container with ID starting with d08809787a646c93af3a23ff64b1842dc624b0dab691eb98f5e8f705e19832e4 not found: ID does not exist"
Jul 07 02:38:15.339663 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339583    2566 scope.go:117] "RemoveContainer" containerID="ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"
Jul 07 02:38:15.339838 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339819    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee"} err="failed to get container status \"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee\": rpc error: code = NotFound desc = could not find container \"ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee\": container with ID starting with ada9b117bda5828e06bdfacb4ec27e2c1f2155c0741b893fac4ac6b9b3da3aee not found: ID does not exist"
Jul 07 02:38:15.339887 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.339839    2566 scope.go:117] "RemoveContainer" containerID="452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"
Jul 07 02:38:15.340047 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.340029    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec"} err="failed to get container status \"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec\": rpc error: code = NotFound desc = could not find container \"452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec\": container with ID starting with 452c0efb0878737ca4083789406efeeb4387f95764c5cf848eea2bccf70203ec not found: ID does not exist"
Jul 07 02:38:15.364359 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kserve-provision-location\") pod \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") "
Jul 07 02:38:15.364458 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-cabundle-cert\") pod \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") "
Jul 07 02:38:15.364458 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364403    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls\") pod \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") "
Jul 07 02:38:15.364546 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364480    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-rbac-proxy-sar-config\") pod \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") "
Jul 07 02:38:15.364546 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364538    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdt76\" (UniqueName: \"kubernetes.io/projected/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-api-access-xdt76\") pod \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\" (UID: \"a2bcd02a-a85c-4f08-aa68-b6bf85e02933\") "
Jul 07 02:38:15.364789 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364753    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2bcd02a-a85c-4f08-aa68-b6bf85e02933" (UID: "a2bcd02a-a85c-4f08-aa68-b6bf85e02933"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:38:15.364921 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.364899    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2bcd02a-a85c-4f08-aa68-b6bf85e02933" (UID: "a2bcd02a-a85c-4f08-aa68-b6bf85e02933"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:38:15.366452 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.366425    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2bcd02a-a85c-4f08-aa68-b6bf85e02933" (UID: "a2bcd02a-a85c-4f08-aa68-b6bf85e02933"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:38:15.366647 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.366619    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-api-access-xdt76" (OuterVolumeSpecName: "kube-api-access-xdt76") pod "a2bcd02a-a85c-4f08-aa68-b6bf85e02933" (UID: "a2bcd02a-a85c-4f08-aa68-b6bf85e02933"). InnerVolumeSpecName "kube-api-access-xdt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:38:15.369094 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.369072    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2bcd02a-a85c-4f08-aa68-b6bf85e02933" (UID: "a2bcd02a-a85c-4f08-aa68-b6bf85e02933"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:15.465377 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.465335    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdt76\" (UniqueName: \"kubernetes.io/projected/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-api-access-xdt76\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:38:15.465377 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.465369    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:38:15.465377 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.465379    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:38:15.465377 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.465389    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:38:15.465887 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.465398    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2bcd02a-a85c-4f08-aa68-b6bf85e02933-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:38:15.637741 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.637703    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"]
Jul 07 02:38:15.641801 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:15.641771    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-p7ntg"]
Jul 07 02:38:16.848768 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:16.848734    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" path="/var/lib/kubelet/pods/a2bcd02a-a85c-4f08-aa68-b6bf85e02933/volumes"
Jul 07 02:38:19.587687 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.587639    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"]
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.587964    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kserve-container"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.587975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kserve-container"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.587990    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="storage-initializer"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.587996    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="storage-initializer"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.588004    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kube-rbac-proxy"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.588010    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kube-rbac-proxy"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.588057    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kube-rbac-proxy"
Jul 07 02:38:19.588076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.588065    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2bcd02a-a85c-4f08-aa68-b6bf85e02933" containerName="kserve-container"
Jul 07 02:38:19.591621 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.591596    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.594619 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.594593    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 07 02:38:19.594787 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.594628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 07 02:38:19.595649 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.595635    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:38:19.595963 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.595947    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:38:19.596234 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.596217    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:38:19.596296 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.596251    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76zzm\""
Jul 07 02:38:19.596906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.596883    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:38:19.606865 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.606837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"]
Jul 07 02:38:19.698535 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.698495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.698718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.698552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wvn\" (UniqueName: \"kubernetes.io/projected/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-api-access-n6wvn\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.698718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.698615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.698718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.698654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.698718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.698707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.799756 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.799714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.799928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.799769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wvn\" (UniqueName: \"kubernetes.io/projected/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-api-access-n6wvn\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.799928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.799805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.799928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.799826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.799928 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.799846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.799928 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:38:19.799882    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 07 02:38:19.800220 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:38:19.799979    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls podName:c3d54faf-4e20-49ab-b5d4-a15c261f0fbf nodeName:}" failed. No retries permitted until 2026-07-07 02:38:20.299940479 +0000 UTC m=+1235.959049235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" (UID: "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 07 02:38:19.800291 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.800245    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.800517 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.800500    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.800553 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.800538    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:19.808960 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:19.808935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wvn\" (UniqueName: \"kubernetes.io/projected/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-api-access-n6wvn\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:20.305082 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:20.305036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:20.307511 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:20.307488    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:20.502297 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:20.502269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:20.636726 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:20.636695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"]
Jul 07 02:38:20.640306 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:38:20.640250    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d54faf_4e20_49ab_b5d4_a15c261f0fbf.slice/crio-2e12abe7755a2dcdf12e1f6144210c73021bbe5536bbbef9c33277a8145e22a4 WatchSource:0}: Error finding container 2e12abe7755a2dcdf12e1f6144210c73021bbe5536bbbef9c33277a8145e22a4: Status 404 returned error can't find the container with id 2e12abe7755a2dcdf12e1f6144210c73021bbe5536bbbef9c33277a8145e22a4
Jul 07 02:38:21.343819 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:21.343787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerStarted","Data":"944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd"}
Jul 07 02:38:21.344009 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:21.343848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerStarted","Data":"2e12abe7755a2dcdf12e1f6144210c73021bbe5536bbbef9c33277a8145e22a4"}
Jul 07 02:38:26.368059 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:26.368025    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerID="944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd" exitCode=0
Jul 07 02:38:26.368401 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:26.368096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerDied","Data":"944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd"}
Jul 07 02:38:27.375616 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:27.375579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerStarted","Data":"75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671"}
Jul 07 02:38:27.376003 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:27.375642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerStarted","Data":"0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73"}
Jul 07 02:38:27.376003 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:27.375851    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:27.401841 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:27.401793    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" podStartSLOduration=3.450846255 podStartE2EDuration="8.401779944s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.950933689s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:27.400289132 +0000 UTC m=+1243.059397900" watchObservedRunningTime="2026-07-07 02:38:27.401779944 +0000 UTC m=+1243.060888709"
Jul 07 02:38:28.380745 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:28.380708    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:38:34.389702 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:38:34.389655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:40:01.847699 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:01.847653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:40:09.715660 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:09.715582    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"]
Jul 07 02:40:09.716221 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:09.715904    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kserve-container" containerID="cri-o://0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73" gracePeriod=30
Jul 07 02:40:09.716221 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:09.715917    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kube-rbac-proxy" containerID="cri-o://75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671" gracePeriod=30
Jul 07 02:40:09.838599 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:09.838570    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerID="75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671" exitCode=2
Jul 07 02:40:09.838779 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:09.838643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerDied","Data":"75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671"}
Jul 07 02:40:10.674546 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.674517    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:40:10.799296 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kserve-provision-location\") pod \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") "
Jul 07 02:40:10.799296 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799266    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls\") pod \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") "
Jul 07 02:40:10.799848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-cabundle-cert\") pod \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") "
Jul 07 02:40:10.799848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6wvn\" (UniqueName: \"kubernetes.io/projected/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-api-access-n6wvn\") pod \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") "
Jul 07 02:40:10.799848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-rbac-proxy-sar-config\") pod \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\" (UID: \"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf\") "
Jul 07 02:40:10.799848 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799788    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" (UID: "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:40:10.800061 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.799920    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" (UID: "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:40:10.801539 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.801511    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-api-access-n6wvn" (OuterVolumeSpecName: "kube-api-access-n6wvn") pod "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" (UID: "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf"). InnerVolumeSpecName "kube-api-access-n6wvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:40:10.801686 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.801650    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" (UID: "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:40:10.804027 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.803994    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" (UID: "c3d54faf-4e20-49ab-b5d4-a15c261f0fbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:40:10.845662 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.845629    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerID="0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73" exitCode=0
Jul 07 02:40:10.845845 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.845778    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"
Jul 07 02:40:10.847805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.847778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerDied","Data":"0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73"}
Jul 07 02:40:10.847939 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.847829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8" event={"ID":"c3d54faf-4e20-49ab-b5d4-a15c261f0fbf","Type":"ContainerDied","Data":"2e12abe7755a2dcdf12e1f6144210c73021bbe5536bbbef9c33277a8145e22a4"}
Jul 07 02:40:10.847939 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.847856    2566 scope.go:117] "RemoveContainer" containerID="75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671"
Jul 07 02:40:10.859932 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.859909    2566 scope.go:117] "RemoveContainer" containerID="0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73"
Jul 07 02:40:10.868504 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.868474    2566 scope.go:117] "RemoveContainer" containerID="944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd"
Jul 07 02:40:10.875480 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.875449    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"]
Jul 07 02:40:10.876636 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.876616    2566 scope.go:117] "RemoveContainer" containerID="75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671"
Jul 07 02:40:10.876968 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:40:10.876949    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671\": container with ID starting with 75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671 not found: ID does not exist" containerID="75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671"
Jul 07 02:40:10.877044 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.876980    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671"} err="failed to get container status \"75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671\": rpc error: code = NotFound desc = could not find container \"75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671\": container with ID starting with 75ac89fe0c5e717cb52f61e2a78814da1cdfd19f966e0a4855e6608fdd40e671 not found: ID does not exist"
Jul 07 02:40:10.877044 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.877007    2566 scope.go:117] "RemoveContainer" containerID="0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73"
Jul 07 02:40:10.877274 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:40:10.877256    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73\": container with ID starting with 0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73 not found: ID does not exist" containerID="0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73"
Jul 07 02:40:10.877337 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.877278    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73"} err="failed to get container status \"0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73\": rpc error: code = NotFound desc = could not find container \"0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73\": container with ID starting with 0ea37be3ff41270590d3fa22cab1059917acc2ea22a058d3959398caabfeae73 not found: ID does not exist"
Jul 07 02:40:10.877337 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.877293    2566 scope.go:117] "RemoveContainer" containerID="944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd"
Jul 07 02:40:10.877582 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:40:10.877559    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd\": container with ID starting with 944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd not found: ID does not exist" containerID="944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd"
Jul 07 02:40:10.877686 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.877589    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd"} err="failed to get container status \"944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd\": rpc error: code = NotFound desc = could not find container \"944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd\": container with ID starting with 944a10a60d08f33587824fa604a68536c29988f270e35da000ea40bbd75d0cfd not found: ID does not exist"
Jul 07 02:40:10.879918 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.879890    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrh8"]
Jul 07 02:40:10.900342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.900308    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:40:10.900342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.900339    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:40:10.900515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.900354    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:40:10.900515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.900363    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:40:10.900515 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:10.900373    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wvn\" (UniqueName: \"kubernetes.io/projected/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf-kube-api-access-n6wvn\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:40:12.847261 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:12.847228    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" path="/var/lib/kubelet/pods/c3d54faf-4e20-49ab-b5d4-a15c261f0fbf/volumes"
Jul 07 02:40:14.914929 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.914895    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"]
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915215    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="storage-initializer"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915226    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="storage-initializer"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kube-rbac-proxy"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915250    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kube-rbac-proxy"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915258    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kserve-container"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915263    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kserve-container"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915316    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kube-rbac-proxy"
Jul 07 02:40:14.915342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.915327    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d54faf-4e20-49ab-b5d4-a15c261f0fbf" containerName="kserve-container"
Jul 07 02:40:14.918437 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.918416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:14.920997 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.920962    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 07 02:40:14.921131 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.920962    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 07 02:40:14.921131 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.921068    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:40:14.921284 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.921252    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:40:14.921846 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.921828    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:40:14.921937 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.921861    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:40:14.922122 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.922106    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76zzm\""
Jul 07 02:40:14.929311 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:14.929281    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"]
Jul 07 02:40:15.036249 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.036212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cxv\" (UniqueName: \"kubernetes.io/projected/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-api-access-c5cxv\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.036249 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.036243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea477b2-a834-44c5-95a2-4f44f02979d5-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.036457 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.036275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.036457 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.036351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea477b2-a834-44c5-95a2-4f44f02979d5-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.036457 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.036391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.136992 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.136963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137176 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.136998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cxv\" (UniqueName: \"kubernetes.io/projected/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-api-access-c5cxv\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137176 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.137021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea477b2-a834-44c5-95a2-4f44f02979d5-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137176 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.137053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137176 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.137095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea477b2-a834-44c5-95a2-4f44f02979d5-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137527 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.137500    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea477b2-a834-44c5-95a2-4f44f02979d5-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137770 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.137752    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.137844 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.137772    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.139654 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.139625    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea477b2-a834-44c5-95a2-4f44f02979d5-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.145444 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.145408    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cxv\" (UniqueName: \"kubernetes.io/projected/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-api-access-c5cxv\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.230157 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.230111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:15.366023 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.365998    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"]
Jul 07 02:40:15.368246 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:40:15.368209    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea477b2_a834_44c5_95a2_4f44f02979d5.slice/crio-6e73c3ace460b3341305f5474a3c3bfd68cd03a0832ad75061427c17b45b0d0f WatchSource:0}: Error finding container 6e73c3ace460b3341305f5474a3c3bfd68cd03a0832ad75061427c17b45b0d0f: Status 404 returned error can't find the container with id 6e73c3ace460b3341305f5474a3c3bfd68cd03a0832ad75061427c17b45b0d0f
Jul 07 02:40:15.872191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.872151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerStarted","Data":"de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e"}
Jul 07 02:40:15.872344 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:15.872222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerStarted","Data":"6e73c3ace460b3341305f5474a3c3bfd68cd03a0832ad75061427c17b45b0d0f"}
Jul 07 02:40:46.005014 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:46.004967    2566 generic.go:356] "Generic (PLEG): container finished" podID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerID="de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e" exitCode=0
Jul 07 02:40:46.005461 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:46.005043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerDied","Data":"de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e"}
Jul 07 02:40:47.012167 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:47.012133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerStarted","Data":"b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4"}
Jul 07 02:40:47.012565 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:47.012199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerStarted","Data":"4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3"}
Jul 07 02:40:47.012565 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:47.012531    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:47.012642 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:47.012589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:40:47.036684 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:47.036608    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" podStartSLOduration=3.412202868 podStartE2EDuration="33.036589896s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.624387028s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:40:47.036032216 +0000 UTC m=+1382.695140979" watchObservedRunningTime="2026-07-07 02:40:47.036589896 +0000 UTC m=+1382.695698660"
Jul 07 02:40:53.023578 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:40:53.023551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:42:23.027694 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:23.027643    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:42:25.101532 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:25.101495    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"]
Jul 07 02:42:25.101994 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:25.101760    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kserve-container" containerID="cri-o://4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3" gracePeriod=30
Jul 07 02:42:25.101994 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:25.101835    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kube-rbac-proxy" containerID="cri-o://b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4" gracePeriod=30
Jul 07 02:42:25.463686 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:25.463634    2566 generic.go:356] "Generic (PLEG): container finished" podID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerID="b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4" exitCode=2
Jul 07 02:42:25.463858 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:25.463718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerDied","Data":"b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4"}
Jul 07 02:42:25.947783 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:25.947751    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:42:26.031529 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.031495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea477b2-a834-44c5-95a2-4f44f02979d5-proxy-tls\") pod \"eea477b2-a834-44c5-95a2-4f44f02979d5\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") "
Jul 07 02:42:26.031529 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.031544    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea477b2-a834-44c5-95a2-4f44f02979d5-kserve-provision-location\") pod \"eea477b2-a834-44c5-95a2-4f44f02979d5\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") "
Jul 07 02:42:26.031805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.031578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5cxv\" (UniqueName: \"kubernetes.io/projected/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-api-access-c5cxv\") pod \"eea477b2-a834-44c5-95a2-4f44f02979d5\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") "
Jul 07 02:42:26.031805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.031633    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-cabundle-cert\") pod \"eea477b2-a834-44c5-95a2-4f44f02979d5\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") "
Jul 07 02:42:26.031805 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.031700    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-rbac-proxy-sar-config\") pod \"eea477b2-a834-44c5-95a2-4f44f02979d5\" (UID: \"eea477b2-a834-44c5-95a2-4f44f02979d5\") "
Jul 07 02:42:26.032027 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.031998    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eea477b2-a834-44c5-95a2-4f44f02979d5" (UID: "eea477b2-a834-44c5-95a2-4f44f02979d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:42:26.032027 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.032017    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eea477b2-a834-44c5-95a2-4f44f02979d5" (UID: "eea477b2-a834-44c5-95a2-4f44f02979d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:42:26.033926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.033884    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea477b2-a834-44c5-95a2-4f44f02979d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eea477b2-a834-44c5-95a2-4f44f02979d5" (UID: "eea477b2-a834-44c5-95a2-4f44f02979d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:42:26.033926 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.033910    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-api-access-c5cxv" (OuterVolumeSpecName: "kube-api-access-c5cxv") pod "eea477b2-a834-44c5-95a2-4f44f02979d5" (UID: "eea477b2-a834-44c5-95a2-4f44f02979d5"). InnerVolumeSpecName "kube-api-access-c5cxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:42:26.036129 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.036084    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea477b2-a834-44c5-95a2-4f44f02979d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eea477b2-a834-44c5-95a2-4f44f02979d5" (UID: "eea477b2-a834-44c5-95a2-4f44f02979d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:42:26.132367 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.132329    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea477b2-a834-44c5-95a2-4f44f02979d5-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:42:26.132367 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.132365    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea477b2-a834-44c5-95a2-4f44f02979d5-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:42:26.132367 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.132376    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cxv\" (UniqueName: \"kubernetes.io/projected/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-api-access-c5cxv\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:42:26.132814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.132385    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:42:26.132814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.132394    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea477b2-a834-44c5-95a2-4f44f02979d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:42:26.469317 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.469280    2566 generic.go:356] "Generic (PLEG): container finished" podID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerID="4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3" exitCode=0
Jul 07 02:42:26.469488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.469343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"
Jul 07 02:42:26.469488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.469366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerDied","Data":"4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3"}
Jul 07 02:42:26.469488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.469436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr" event={"ID":"eea477b2-a834-44c5-95a2-4f44f02979d5","Type":"ContainerDied","Data":"6e73c3ace460b3341305f5474a3c3bfd68cd03a0832ad75061427c17b45b0d0f"}
Jul 07 02:42:26.469488 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.469464    2566 scope.go:117] "RemoveContainer" containerID="b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4"
Jul 07 02:42:26.479788 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.479661    2566 scope.go:117] "RemoveContainer" containerID="4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3"
Jul 07 02:42:26.488599 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.488571    2566 scope.go:117] "RemoveContainer" containerID="de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e"
Jul 07 02:42:26.492822 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.492792    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"]
Jul 07 02:42:26.498050 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.498017    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-bg2xr"]
Jul 07 02:42:26.498998 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.498978    2566 scope.go:117] "RemoveContainer" containerID="b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4"
Jul 07 02:42:26.499345 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:42:26.499323    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4\": container with ID starting with b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4 not found: ID does not exist" containerID="b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4"
Jul 07 02:42:26.499388 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.499358    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4"} err="failed to get container status \"b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4\": rpc error: code = NotFound desc = could not find container \"b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4\": container with ID starting with b1263420aa250981689b5228f620b5318306952e64dcde50da6ddfb47c963df4 not found: ID does not exist"
Jul 07 02:42:26.499388 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.499378    2566 scope.go:117] "RemoveContainer" containerID="4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3"
Jul 07 02:42:26.499663 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:42:26.499646    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3\": container with ID starting with 4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3 not found: ID does not exist" containerID="4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3"
Jul 07 02:42:26.499736 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.499691    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3"} err="failed to get container status \"4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3\": rpc error: code = NotFound desc = could not find container \"4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3\": container with ID starting with 4697e75d03ef15d9f4ae8f70c6b3fc6da22b5a1ed7a1b3119d3d27ab7c05e8c3 not found: ID does not exist"
Jul 07 02:42:26.499736 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.499708    2566 scope.go:117] "RemoveContainer" containerID="de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e"
Jul 07 02:42:26.499949 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:42:26.499931    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e\": container with ID starting with de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e not found: ID does not exist" containerID="de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e"
Jul 07 02:42:26.500123 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.499953    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e"} err="failed to get container status \"de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e\": rpc error: code = NotFound desc = could not find container \"de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e\": container with ID starting with de720b22981d7fe1845fb334a155d5c156f02504ac69788270f9778d7a75994e not found: ID does not exist"
Jul 07 02:42:26.847263 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:26.847182    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" path="/var/lib/kubelet/pods/eea477b2-a834-44c5-95a2-4f44f02979d5/volumes"
Jul 07 02:42:30.235141 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235102    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"]
Jul 07 02:42:30.235558 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235533    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="storage-initializer"
Jul 07 02:42:30.235558 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235557    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="storage-initializer"
Jul 07 02:42:30.235758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235579    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kserve-container"
Jul 07 02:42:30.235758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235586    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kserve-container"
Jul 07 02:42:30.235758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235597    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kube-rbac-proxy"
Jul 07 02:42:30.235758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235603    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kube-rbac-proxy"
Jul 07 02:42:30.235758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235662    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kserve-container"
Jul 07 02:42:30.235758 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.235689    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea477b2-a834-44c5-95a2-4f44f02979d5" containerName="kube-rbac-proxy"
Jul 07 02:42:30.239915 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.239896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.242366 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.242340    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:42:30.242483 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.242339    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 07 02:42:30.242483 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.242408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 07 02:42:30.242483 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.242454    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:42:30.242483 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.242465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-76zzm\""
Jul 07 02:42:30.242728 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.242412    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:42:30.243219 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.243196    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:42:30.249832 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.249811    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"]
Jul 07 02:42:30.366535 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.366500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.366535 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.366546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.366814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.366569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.366814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.366617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtcg\" (UniqueName: \"kubernetes.io/projected/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-api-access-rgtcg\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.366814 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.366638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.467325 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.467286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtcg\" (UniqueName: \"kubernetes.io/projected/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-api-access-rgtcg\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.467325 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.467330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.467530 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.467374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.467530 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.467397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.467530 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.467417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.467820 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.467801    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.468231 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.468207    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.468320 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.468240    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.470088 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.470061    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.474879 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.474850    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtcg\" (UniqueName: \"kubernetes.io/projected/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-api-access-rgtcg\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.552730 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.552611    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:30.685996 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:30.685969    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"]
Jul 07 02:42:30.688459 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:42:30.688427    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc0c2e3_17ed_42f3_b58c_5df0c05532b9.slice/crio-c7d35d83e662387ecba1f0840bc40d3db5b62f5f64664e8fa2d4c32f2db013ff WatchSource:0}: Error finding container c7d35d83e662387ecba1f0840bc40d3db5b62f5f64664e8fa2d4c32f2db013ff: Status 404 returned error can't find the container with id c7d35d83e662387ecba1f0840bc40d3db5b62f5f64664e8fa2d4c32f2db013ff
Jul 07 02:42:31.494031 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:31.493987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerStarted","Data":"c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69"}
Jul 07 02:42:31.494506 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:31.494071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerStarted","Data":"c7d35d83e662387ecba1f0840bc40d3db5b62f5f64664e8fa2d4c32f2db013ff"}
Jul 07 02:42:40.534450 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:40.534418    2566 generic.go:356] "Generic (PLEG): container finished" podID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerID="c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69" exitCode=0
Jul 07 02:42:40.534849 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:40.534495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerDied","Data":"c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69"}
Jul 07 02:42:41.541786 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:41.541744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerStarted","Data":"dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc"}
Jul 07 02:42:41.542169 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:41.541831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerStarted","Data":"a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477"}
Jul 07 02:42:41.542169 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:41.542057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:41.563617 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:41.563570    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" podStartSLOduration=2.190513205 podStartE2EDuration="11.563554452s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.373041247s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:42:41.561637003 +0000 UTC m=+1497.220745822" watchObservedRunningTime="2026-07-07 02:42:41.563554452 +0000 UTC m=+1497.222663213"
Jul 07 02:42:42.546457 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:42.546427    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:42.547944 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:42.547913    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 02:42:43.551548 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:43.551502    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 02:42:44.823314 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:44.823283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:42:44.825657 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:44.825631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:42:44.831172 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:44.831143    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:42:44.833437 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:44.833420    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:42:48.556489 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:48.556450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:42:48.557183 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:48.557156    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 07 02:42:58.558470 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:42:58.558429    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:43:00.311467 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:00.311435    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"]
Jul 07 02:43:00.311986 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:00.311720    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container" containerID="cri-o://a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477" gracePeriod=30
Jul 07 02:43:00.311986 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:00.311749    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kube-rbac-proxy" containerID="cri-o://dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc" gracePeriod=30
Jul 07 02:43:00.635270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:00.635168    2566 generic.go:356] "Generic (PLEG): container finished" podID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerID="dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc" exitCode=2
Jul 07 02:43:00.635270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:00.635243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerDied","Data":"dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc"}
Jul 07 02:43:01.273728 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.273702    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:43:01.321315 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321283    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kserve-provision-location\") pod \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") "
Jul 07 02:43:01.321793 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-proxy-tls\") pod \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") "
Jul 07 02:43:01.321793 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321406    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-cabundle-cert\") pod \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") "
Jul 07 02:43:01.321793 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321465    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-rbac-proxy-sar-config\") pod \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") "
Jul 07 02:43:01.321793 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321512    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgtcg\" (UniqueName: \"kubernetes.io/projected/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-api-access-rgtcg\") pod \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\" (UID: \"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9\") "
Jul 07 02:43:01.322162 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321870    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" (UID: "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:43:01.322162 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.321867    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" (UID: "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:43:01.322386 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.322360    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" (UID: "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:43:01.323462 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.323439    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" (UID: "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:43:01.323619 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.323597    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-api-access-rgtcg" (OuterVolumeSpecName: "kube-api-access-rgtcg") pod "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" (UID: "4cc0c2e3-17ed-42f3-b58c-5df0c05532b9"). InnerVolumeSpecName "kube-api-access-rgtcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:43:01.422871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.422760    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:43:01.422871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.422808    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtcg\" (UniqueName: \"kubernetes.io/projected/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kube-api-access-rgtcg\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:43:01.422871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.422818    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-kserve-provision-location\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:43:01.422871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.422829    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-proxy-tls\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:43:01.422871 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.422838    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9-cabundle-cert\") on node \"ip-10-0-142-238.ec2.internal\" DevicePath \"\""
Jul 07 02:43:01.641897 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.641864    2566 generic.go:356] "Generic (PLEG): container finished" podID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerID="a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477" exitCode=0
Jul 07 02:43:01.642076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.641924    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"
Jul 07 02:43:01.642076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.641941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerDied","Data":"a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477"}
Jul 07 02:43:01.642076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.641989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw" event={"ID":"4cc0c2e3-17ed-42f3-b58c-5df0c05532b9","Type":"ContainerDied","Data":"c7d35d83e662387ecba1f0840bc40d3db5b62f5f64664e8fa2d4c32f2db013ff"}
Jul 07 02:43:01.642076 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.642014    2566 scope.go:117] "RemoveContainer" containerID="dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc"
Jul 07 02:43:01.651231 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.651212    2566 scope.go:117] "RemoveContainer" containerID="a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477"
Jul 07 02:43:01.658961 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.658939    2566 scope.go:117] "RemoveContainer" containerID="c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69"
Jul 07 02:43:01.663836 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.663810    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"]
Jul 07 02:43:01.667659 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.667633    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-npbzw"]
Jul 07 02:43:01.668033 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.668012    2566 scope.go:117] "RemoveContainer" containerID="dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc"
Jul 07 02:43:01.668359 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:43:01.668341    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc\": container with ID starting with dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc not found: ID does not exist" containerID="dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc"
Jul 07 02:43:01.668422 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.668368    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc"} err="failed to get container status \"dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc\": rpc error: code = NotFound desc = could not find container \"dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc\": container with ID starting with dde81984e29e7cdc99697b829625e8443653a168768269ee435d2c3a68b2e1bc not found: ID does not exist"
Jul 07 02:43:01.668422 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.668386    2566 scope.go:117] "RemoveContainer" containerID="a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477"
Jul 07 02:43:01.668735 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:43:01.668654    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477\": container with ID starting with a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477 not found: ID does not exist" containerID="a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477"
Jul 07 02:43:01.668735 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.668686    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477"} err="failed to get container status \"a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477\": rpc error: code = NotFound desc = could not find container \"a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477\": container with ID starting with a7059bf4a69307a6eabe4305789144f6d876b2b0017418341b60e6a400d6a477 not found: ID does not exist"
Jul 07 02:43:01.668735 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.668699    2566 scope.go:117] "RemoveContainer" containerID="c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69"
Jul 07 02:43:01.668981 ip-10-0-142-238 kubenswrapper[2566]: E0707 02:43:01.668940    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69\": container with ID starting with c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69 not found: ID does not exist" containerID="c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69"
Jul 07 02:43:01.668981 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:01.668959    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69"} err="failed to get container status \"c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69\": rpc error: code = NotFound desc = could not find container \"c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69\": container with ID starting with c6265da60ef030e9d7a57a46b3285fcffa96baf88e029d45b6f221d80cd94d69 not found: ID does not exist"
Jul 07 02:43:02.848140 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:02.848109    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" path="/var/lib/kubelet/pods/4cc0c2e3-17ed-42f3-b58c-5df0c05532b9/volumes"
Jul 07 02:43:34.212442 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212406    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4w788/must-gather-jr4pf"]
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212708    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="storage-initializer"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212720    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="storage-initializer"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212730    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212735    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212750    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kube-rbac-proxy"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212755    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kube-rbac-proxy"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212819    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kube-rbac-proxy"
Jul 07 02:43:34.212906 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.212828    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc0c2e3-17ed-42f3-b58c-5df0c05532b9" containerName="kserve-container"
Jul 07 02:43:34.215876 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.215857    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.219395 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.219371    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4w788\"/\"default-dockercfg-5l75c\""
Jul 07 02:43:34.219508 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.219393    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4w788\"/\"openshift-service-ca.crt\""
Jul 07 02:43:34.220396 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.220378    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4w788\"/\"kube-root-ca.crt\""
Jul 07 02:43:34.225755 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.225735    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w788/must-gather-jr4pf"]
Jul 07 02:43:34.403662 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.403627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqwj\" (UniqueName: \"kubernetes.io/projected/3982045b-c302-409f-b0d6-35e36877ab0f-kube-api-access-6jqwj\") pod \"must-gather-jr4pf\" (UID: \"3982045b-c302-409f-b0d6-35e36877ab0f\") " pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.403849 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.403745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3982045b-c302-409f-b0d6-35e36877ab0f-must-gather-output\") pod \"must-gather-jr4pf\" (UID: \"3982045b-c302-409f-b0d6-35e36877ab0f\") " pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.504501 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.504408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqwj\" (UniqueName: \"kubernetes.io/projected/3982045b-c302-409f-b0d6-35e36877ab0f-kube-api-access-6jqwj\") pod \"must-gather-jr4pf\" (UID: \"3982045b-c302-409f-b0d6-35e36877ab0f\") " pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.504501 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.504474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3982045b-c302-409f-b0d6-35e36877ab0f-must-gather-output\") pod \"must-gather-jr4pf\" (UID: \"3982045b-c302-409f-b0d6-35e36877ab0f\") " pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.504861 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.504839    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3982045b-c302-409f-b0d6-35e36877ab0f-must-gather-output\") pod \"must-gather-jr4pf\" (UID: \"3982045b-c302-409f-b0d6-35e36877ab0f\") " pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.513048 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.513024    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqwj\" (UniqueName: \"kubernetes.io/projected/3982045b-c302-409f-b0d6-35e36877ab0f-kube-api-access-6jqwj\") pod \"must-gather-jr4pf\" (UID: \"3982045b-c302-409f-b0d6-35e36877ab0f\") " pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.525964 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.525942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w788/must-gather-jr4pf"
Jul 07 02:43:34.650241 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.650131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w788/must-gather-jr4pf"]
Jul 07 02:43:34.653006 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:43:34.652977    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3982045b_c302_409f_b0d6_35e36877ab0f.slice/crio-8e29bdec15c23e4e8f71fac9ab573346a0f8f358a2253b78dfb3fe312991e225 WatchSource:0}: Error finding container 8e29bdec15c23e4e8f71fac9ab573346a0f8f358a2253b78dfb3fe312991e225: Status 404 returned error can't find the container with id 8e29bdec15c23e4e8f71fac9ab573346a0f8f358a2253b78dfb3fe312991e225
Jul 07 02:43:34.655141 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.655126    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:43:34.791799 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:34.791719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w788/must-gather-jr4pf" event={"ID":"3982045b-c302-409f-b0d6-35e36877ab0f","Type":"ContainerStarted","Data":"8e29bdec15c23e4e8f71fac9ab573346a0f8f358a2253b78dfb3fe312991e225"}
Jul 07 02:43:35.797999 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:35.797964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w788/must-gather-jr4pf" event={"ID":"3982045b-c302-409f-b0d6-35e36877ab0f","Type":"ContainerStarted","Data":"a19b342424c05161e89d60cbb0659cb0f728a155e4293a1c0021de5262a92665"}
Jul 07 02:43:35.798461 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:35.798028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w788/must-gather-jr4pf" event={"ID":"3982045b-c302-409f-b0d6-35e36877ab0f","Type":"ContainerStarted","Data":"4de0be8fd6e5a69b94e40281f79adfdc7deeca8517ee92348af67af1a6128a2c"}
Jul 07 02:43:35.814092 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:35.814036    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4w788/must-gather-jr4pf" podStartSLOduration=1.126520466 podStartE2EDuration="1.814014648s" totalImagesPullingTime="687.494182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:43:35.811896771 +0000 UTC m=+1551.471005534" watchObservedRunningTime="2026-07-07 02:43:35.814014648 +0000 UTC m=+1551.473123411"
Jul 07 02:43:36.727631 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:36.727600    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gvjbd_c3cee6f8-980e-4530-b615-b02786a8e117/global-pull-secret-syncer/0.log"
Jul 07 02:43:36.874940 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:36.874908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lv8qk_d0a8096f-0d4c-45c5-a548-316323d7dc8a/konnectivity-agent/0.log"
Jul 07 02:43:36.971491 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:36.971459    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-238.ec2.internal_4b0b802c6511aef6dcda6b6d308a1cf9/haproxy/0.log"
Jul 07 02:43:41.709990 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:41.709953    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-5pqn5_4753d121-c72d-45fa-a249-ef8ccc73d525/cluster-monitoring-operator/0.log"
Jul 07 02:43:41.742059 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:41.742030    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-n2tjj_cd612176-f5ee-4c27-b8b1-f2c00b9d5807/kube-state-metrics/0.log"
Jul 07 02:43:41.762979 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:41.762944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-n2tjj_cd612176-f5ee-4c27-b8b1-f2c00b9d5807/kube-rbac-proxy-main/0.log"
Jul 07 02:43:41.809721 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:41.809664    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-n2tjj_cd612176-f5ee-4c27-b8b1-f2c00b9d5807/kube-rbac-proxy-self/0.log"
Jul 07 02:43:41.842058 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:41.842021    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74c499c889-vcbt9_69503df1-69f1-4822-825d-8178b665c64f/metrics-server/0.log"
Jul 07 02:43:41.872227 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:41.872197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-4dsr9_f3bb8f20-0be7-4bfd-8319-86d6007c9b4d/monitoring-plugin/0.log"
Jul 07 02:43:42.072609 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:42.072532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s9hxd_c10710ff-0a3e-41fd-b942-a7146fec1cc3/node-exporter/0.log"
Jul 07 02:43:42.094189 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:42.094155    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s9hxd_c10710ff-0a3e-41fd-b942-a7146fec1cc3/kube-rbac-proxy/0.log"
Jul 07 02:43:42.114898 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:42.114865    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s9hxd_c10710ff-0a3e-41fd-b942-a7146fec1cc3/init-textfile/0.log"
Jul 07 02:43:42.407760 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:42.407641    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-cb6t9_8baa35ce-c17b-4f0c-8b99-620fa133a0d6/prometheus-operator/0.log"
Jul 07 02:43:42.428851 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:42.428817    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-cb6t9_8baa35ce-c17b-4f0c-8b99-620fa133a0d6/kube-rbac-proxy/0.log"
Jul 07 02:43:44.411225 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:44.411194    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/1.log"
Jul 07 02:43:44.420041 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:44.420013    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-bktlx_4b6fc827-c7ca-4b38-8aef-63d8b12166bd/console-operator/2.log"
Jul 07 02:43:45.036923 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.036885    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"]
Jul 07 02:43:45.042138 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.042111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.050872 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.050473    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"]
Jul 07 02:43:45.111559 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.111522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-sys\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.111768 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.111581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-lib-modules\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.111768 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.111690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-podres\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.111768 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.111735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbsx\" (UniqueName: \"kubernetes.io/projected/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-kube-api-access-qmbsx\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.111950 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.111773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-proc\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213156 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-lib-modules\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213339 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-podres\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213339 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbsx\" (UniqueName: \"kubernetes.io/projected/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-kube-api-access-qmbsx\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213339 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-proc\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213339 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213314    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-podres\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213339 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-sys\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213557 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213364    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-proc\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213557 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213314    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-lib-modules\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.213557 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.213386    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-sys\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.222003 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.221969    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbsx\" (UniqueName: \"kubernetes.io/projected/17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb-kube-api-access-qmbsx\") pod \"perf-node-gather-daemonset-bhsx5\" (UID: \"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb\") " pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.356029 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.355996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.528756 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.528643    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"]
Jul 07 02:43:45.534913 ip-10-0-142-238 kubenswrapper[2566]: W0707 02:43:45.534878    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17011e0e_8ef3_40e3_bd6f_c454c9bfb9eb.slice/crio-a5cfc56a0bc0d9bcaebc61f2775c5da3f76661222c887af84b84298e07ef5b5f WatchSource:0}: Error finding container a5cfc56a0bc0d9bcaebc61f2775c5da3f76661222c887af84b84298e07ef5b5f: Status 404 returned error can't find the container with id a5cfc56a0bc0d9bcaebc61f2775c5da3f76661222c887af84b84298e07ef5b5f
Jul 07 02:43:45.867869 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.867770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5" event={"ID":"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb","Type":"ContainerStarted","Data":"dc63c02d4cca4d0caef486387b7090992c04a71483f8df2e9fcd68abf8f28794"}
Jul 07 02:43:45.867869 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.867846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5" event={"ID":"17011e0e-8ef3-40e3-bd6f-c454c9bfb9eb","Type":"ContainerStarted","Data":"a5cfc56a0bc0d9bcaebc61f2775c5da3f76661222c887af84b84298e07ef5b5f"}
Jul 07 02:43:45.868126 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.867955    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:45.886330 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:45.886258    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5" podStartSLOduration=0.886238037 podStartE2EDuration="886.238037ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:43:45.884036014 +0000 UTC m=+1561.543144780" watchObservedRunningTime="2026-07-07 02:43:45.886238037 +0000 UTC m=+1561.545346800"
Jul 07 02:43:46.483041 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:46.483003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wttkz_c43f4893-fdbb-422f-af88-799cb1e33ec6/dns/0.log"
Jul 07 02:43:46.507270 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:46.507237    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wttkz_c43f4893-fdbb-422f-af88-799cb1e33ec6/kube-rbac-proxy/0.log"
Jul 07 02:43:46.583342 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:46.583311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r9mtv_a1868bb6-d73c-4ac4-b26a-cd9e60b97515/dns-node-resolver/0.log"
Jul 07 02:43:47.050057 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:47.050019    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-565bf886d4-98gzv_dfe3208a-0da7-497d-af10-07265cca435e/registry/0.log"
Jul 07 02:43:47.127715 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:47.127656    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvhhh_cc440849-966b-4c47-8e20-36454af21e2c/node-ca/0.log"
Jul 07 02:43:48.296753 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:48.296717    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-br5cd_61893ef3-a907-4929-9848-e0fee89fd5d9/serve-healthcheck-canary/0.log"
Jul 07 02:43:48.720728 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:48.720646    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-4mqhh_f975e7d9-c1a7-41bd-89fb-6fe6f706aff6/insights-operator/1.log"
Jul 07 02:43:48.720899 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:48.720696    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-4mqhh_f975e7d9-c1a7-41bd-89fb-6fe6f706aff6/insights-operator/0.log"
Jul 07 02:43:48.868009 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:48.867978    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-swmdm_bffe9839-af0d-4f05-8fe0-f2d8bb7fc446/kube-rbac-proxy/0.log"
Jul 07 02:43:48.887435 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:48.887408    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-swmdm_bffe9839-af0d-4f05-8fe0-f2d8bb7fc446/exporter/0.log"
Jul 07 02:43:48.908021 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:48.907989    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-swmdm_bffe9839-af0d-4f05-8fe0-f2d8bb7fc446/extractor/0.log"
Jul 07 02:43:50.957351 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:50.957287    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-f8f7445f4-94lch_dbef2b15-dc5b-41ee-b0ca-b16c490d2464/manager/0.log"
Jul 07 02:43:51.001497 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.001466    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-bbncc_78c74d5a-5437-4fee-98e2-ca2ed8116469/server/0.log"
Jul 07 02:43:51.112763 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.112733    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-swrch_2b9a881b-14bf-478a-ac7e-71eb3213a547/manager/0.log"
Jul 07 02:43:51.156261 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.156233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5f4kr_8725c2b4-4c3e-4c9d-a340-cd729e65b7de/s3-init/0.log"
Jul 07 02:43:51.171968 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.171942    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5f4kr_8725c2b4-4c3e-4c9d-a340-cd729e65b7de/download-hf-model/0.log"
Jul 07 02:43:51.204820 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.204797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-ggq4g_010e7297-9c37-4087-b4f8-610459c09340/s3-tls-init-custom/0.log"
Jul 07 02:43:51.220884 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.220863    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-ggq4g_010e7297-9c37-4087-b4f8-610459c09340/download-hf-model/0.log"
Jul 07 02:43:51.254191 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.254158    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-wxx59_66df9d51-a062-4e90-a128-22f8e42e0cdb/s3-tls-init-serving/0.log"
Jul 07 02:43:51.268591 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.268565    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-wxx59_66df9d51-a062-4e90-a128-22f8e42e0cdb/download-hf-model/0.log"
Jul 07 02:43:51.321361 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.321334    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-nfpl8_643c6f51-c488-4420-9236-3fa5ce33ed5e/seaweedfs-tls-custom/0.log"
Jul 07 02:43:51.886406 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:51.886379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4w788/perf-node-gather-daemonset-bhsx5"
Jul 07 02:43:55.421628 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:55.421530    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-n9t4p_c7544dc0-bff1-42bd-b00d-a8911eee438f/migrator/0.log"
Jul 07 02:43:55.442077 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:55.442045    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-n9t4p_c7544dc0-bff1-42bd-b00d-a8911eee438f/graceful-termination/0.log"
Jul 07 02:43:56.737103 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.737073    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9vrf7_2054e60e-8de7-4424-a6fc-bccb5d0fe9c6/kube-multus/0.log"
Jul 07 02:43:56.763815 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.763787    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/kube-multus-additional-cni-plugins/0.log"
Jul 07 02:43:56.784363 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.784335    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/egress-router-binary-copy/0.log"
Jul 07 02:43:56.805472 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.805442    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/cni-plugins/0.log"
Jul 07 02:43:56.826788 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.826754    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/bond-cni-plugin/0.log"
Jul 07 02:43:56.846718 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.846689    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/routeoverride-cni/0.log"
Jul 07 02:43:56.871398 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.871366    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/whereabouts-cni-bincopy/0.log"
Jul 07 02:43:56.891895 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:56.891869    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2x6b_c245e94b-37ed-45d9-a4aa-b2ddc1e77b6e/whereabouts-cni/0.log"
Jul 07 02:43:57.379652 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:57.379620    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2khl_eb530956-7b82-4d88-975f-3c3e042b7fb8/network-metrics-daemon/0.log"
Jul 07 02:43:57.399513 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:57.399476    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2khl_eb530956-7b82-4d88-975f-3c3e042b7fb8/kube-rbac-proxy/0.log"
Jul 07 02:43:58.402274 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.402235    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-controller/0.log"
Jul 07 02:43:58.420406 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.420356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/0.log"
Jul 07 02:43:58.420406 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.420380    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovn-acl-logging/1.log"
Jul 07 02:43:58.447035 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.446992    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/kube-rbac-proxy-node/0.log"
Jul 07 02:43:58.468802 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.468746    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 02:43:58.489358 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.489330    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/northd/0.log"
Jul 07 02:43:58.509838 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.509811    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/nbdb/0.log"
Jul 07 02:43:58.534000 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.533967    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/sbdb/0.log"
Jul 07 02:43:58.599116 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:58.599082    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g4dxr_327320b3-4c10-453f-811b-48f5033b92ce/ovnkube-controller/0.log"
Jul 07 02:43:59.829101 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:43:59.829062    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ns6nx_bd3f6020-1ea4-4a96-bc08-7d370557b27d/network-check-target-container/0.log"
Jul 07 02:44:00.877317 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:44:00.877285    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-phrvp_af5d8a33-1406-4710-a9ed-3ce2b9e8788b/iptables-alerter/0.log"
Jul 07 02:44:01.530911 ip-10-0-142-238 kubenswrapper[2566]: I0707 02:44:01.530876    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7gkcn_9bc7941d-4a32-488e-9c02-cb43da0a5d04/tuned/0.log"