Jul 01 19:22:14.751925 ip-10-0-131-138 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.224311    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238391    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238410    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238414    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238417    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238420    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238423    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238428    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:15.244490 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238433    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238436    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238439    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238442    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238446    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238449    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238452    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238455    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238457    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238460    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238463    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238469    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238472    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238476    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238478    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238483    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238486    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238489    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:15.259773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238492    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238495    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238504    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238509    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238512    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238516    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238519    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238523    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238526    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238529    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238531    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238534    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238538    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238542    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238545    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238548    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238551    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238554    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238556    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238560    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:15.318242 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238564    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238567    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238570    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238572    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238575    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238577    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238580    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238583    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238587    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238590    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238594    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238598    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238601    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238605    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238608    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238611    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238614    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238618    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238622    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238626    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:15.321678 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238647    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238651    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238654    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238658    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238661    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238664    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238669    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238672    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238674    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238677    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238679    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238684    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238690    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238694    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238697    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238700    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238703    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238708    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238712    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:15.324332 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238715    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238718    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238720    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238724    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238728    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238732    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238734    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238737    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238739    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238742    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238745    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238748    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238757    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.238760    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239600    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239609    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239615    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239619    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239621    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239624    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:15.327903 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239628    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239649    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239652    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239655    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239659    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239663    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239667    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239669    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239674    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239678    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239681    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239685    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239688    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239692    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239695    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239698    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239701    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239704    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239708    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:15.331048 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239712    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239716    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239720    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239724    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239726    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239729    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239733    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239736    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239739    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239741    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239745    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239749    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239752    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239754    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239757    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239759    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239762    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239766    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239771    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:15.334279 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239773    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239776    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239779    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239782    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239786    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239789    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239792    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239795    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239797    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239800    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239803    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239810    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239813    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239815    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239818    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239820    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239824    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239828    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239831    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239834    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:15.336851 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239837    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239840    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239844    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239848    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239851    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239853    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239857    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239860    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239864    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239868    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239870    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239873    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239875    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239878    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239881    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239885    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239889    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239892    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239896    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239900    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:15.340358 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239905    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239909    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239912    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239917    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239920    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239924    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239928    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239931    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239933    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239936    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239939    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239942    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239945    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239949    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.239952    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240383    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240389    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240393    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240395    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240398    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:15.343233 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240401    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240404    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240406    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240410    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240414    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240417    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240419    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240422    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240425    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240427    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240430    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240434    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240437    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240441    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240444    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240451    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240454    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240458    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240460    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240463    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:15.346964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240465    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240468    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240471    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240473    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240477    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240480    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240484    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240488    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240491    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240496    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240498    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240501    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240503    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240506    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240508    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240511    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240517    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240521    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240524    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:15.350151 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240528    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240531    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240533    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240536    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240538    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240542    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240547    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240551    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240557    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240561    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240563    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240566    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240569    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240572    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240576    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240579    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240582    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240584    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240587    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:15.353924 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240590    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240594    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240597    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240601    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240604    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240606    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240609    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240612    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240616    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240618    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240621    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240623    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240625    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240644    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240649    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240652    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240655    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240657    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240660    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240663    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:15.356773 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240666    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:15.356480 ip-10-0-131-138 systemd[1]: Started Kubernetes Kubelet.
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240671    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240675    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240678    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240682    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240687    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240690    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240693    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240695    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240698    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240700    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240703    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240706    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240709    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240713    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.240717    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240794    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240807    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240815    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240820    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240825    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 19:22:15.361228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240828    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240833    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240838    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240843    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240846    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240849    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240853    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240856    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240860    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240864    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240867    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240871    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240875    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240881    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240884    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240888    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240891    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240894    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240900    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240903    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240906    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240909    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240913    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240917    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 19:22:15.364933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240922    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240925    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240929    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240933    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240937    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240942    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240945    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240950    2578 flags.go:64] FLAG: --enable-server="true"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240953    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240959    2578 flags.go:64] FLAG: --event-burst="100"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240962    2578 flags.go:64] FLAG: --event-qps="50"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240965    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240968    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240972    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240978    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240982    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240985    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240988    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240992    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.240996    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241001    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241004    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241007    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241010    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241013    2578 flags.go:64] FLAG: --feature-gates=""
Jul 01 19:22:15.369795 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241019    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241022    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241025    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241028    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241031    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241035    2578 flags.go:64] FLAG: --help="false"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241039    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241043    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241046    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241051    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241056    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241060    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241063    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241066    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241069    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241072    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241076    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241079    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241083    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241086    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241088    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241091    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241095    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241099    2578 flags.go:64] FLAG: --lock-file=""
Jul 01 19:22:15.373527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241102    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241105    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241108    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241115    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241121    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241125    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241129    2578 flags.go:64] FLAG: --logging-format="text"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241132    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241137    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241140    2578 flags.go:64] FLAG: --manifest-url=""
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241143    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241148    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241151    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241155    2578 flags.go:64] FLAG: --max-pods="110"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241158    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241162    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241166    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241171    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241174    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241177    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241181    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241190    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241193    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241197    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 19:22:15.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241202    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241205    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241210    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241213    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241217    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241221    2578 flags.go:64] FLAG: --port="10250"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241225    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241227    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d8fc434a02558a9"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241231    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241234    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241238    2578 flags.go:64] FLAG: --register-node="true"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241242    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241248    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241251    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241255    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241259    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241264    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241267    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241270    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241273    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241277    2578 flags.go:64] FLAG: --runonce="false"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241281    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241285    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241288    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241291    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 19:22:15.381747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241297    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241301    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241304    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241308    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241311    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241314    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241318    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241322    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241325    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241328    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241331    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241338    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241342    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241344    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241348    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241351    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241355    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241359    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241362    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241366    2578 flags.go:64] FLAG: --v="2"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241371    2578 flags.go:64] FLAG: --version="false"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241376    2578 flags.go:64] FLAG: --vmodule=""
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241381    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.241384    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241513    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:15.387528 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241517    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241520    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241523    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241526    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241529    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241532    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241535    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241539    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241541    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241544    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241548    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241552    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241556    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241559    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241562    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241566    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241570    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241574    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241577    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:15.547587 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241580    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241583    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241587    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241591    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241594    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241597    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241599    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241604    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241608    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241610    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241614    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241616    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241620    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241624    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241627    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241646    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241651    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241654    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241657    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241659    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:15.567277 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241663    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241667    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241671    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241675    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241678    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241681    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241683    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241688    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241692    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241695    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241698    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241700    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241703    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241706    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241710    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241712    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241715    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241717    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241720    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241724    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:15.571297 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241728    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241731    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241734    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241737    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241740    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241743    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241747    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241750    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241753    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241755    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241758    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241760    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241765    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241769    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241772    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241774    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241777    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241779    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241782    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241784    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:15.571989 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241788    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241791    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241794    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241796    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241799    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241802    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241804    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241807    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241809    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241811    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241815    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241820    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241823    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241826    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241828    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241831    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241835    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241838    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:15.572679 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.241842    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.242674    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.267818    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.267835    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.268799    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.268811    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268896    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268902    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268909    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268912    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268916    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268919    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268922    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268925    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268927    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:15.573403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268930    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268932    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268938    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268941    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268944    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268947    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268949    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268952    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268954    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268961    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268965    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268967    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268970    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268973    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268975    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268983    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268986    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268988    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268991    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.268994    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:15.573929 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269007    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269010    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269013    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269015    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269018    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269024    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269027    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269030    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269034    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269038    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269040    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269047    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269050    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269053    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269055    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269060    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269063    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269069    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269073    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:15.574794 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269076    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269078    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269081    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269084    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269093    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269096    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269098    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269101    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269103    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269110    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269113    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269115    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269118    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269120    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269133    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269136    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269138    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269141    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269143    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269151    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:15.575400 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269154    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269157    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269159    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269162    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269164    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269172    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269174    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269177    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269179    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269182    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269184    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269191    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269194    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269196    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269199    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269201    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269205    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269207    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269214    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269216    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:15.576102 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269219    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269221    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269224    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269227    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269229    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269237    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269240    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269249    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269257    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269259    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269262    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.269268    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269455    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269461    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269464    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:22:15.576686 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269471    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269474    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269476    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269480    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269485    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269491    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269494    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269497    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269500    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269502    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269505    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269507    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269514    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269517    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269520    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269523    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269525    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269532    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269535    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269537    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:22:15.577213 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269540    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269542    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269546    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269554    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269556    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269565    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269572    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269575    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269579    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269582    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269585    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269591    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269594    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269597    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269599    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269602    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269605    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269611    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269614    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:22:15.577787 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269616    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269619    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269621    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269624    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269626    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269648    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269651    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269655    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269658    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269664    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269668    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269671    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269674    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269677    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269683    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269686    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269689    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269691    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269693    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:22:15.578342 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269707    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269710    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269713    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269716    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269719    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269721    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269728    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269731    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269733    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269736    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269738    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269741    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269743    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269745    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269752    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269755    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269757    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269760    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269762    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269765    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:22:15.578915 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269772    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269775    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269778    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269780    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269783    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269785    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269792    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269795    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269797    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269800    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269802    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269809    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269811    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269820    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269824    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269827    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269834    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:22:15.580750 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:15.269836    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.269841    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.270007    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.272762    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.274622    2578 server.go:1019] "Starting client certificate rotation"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.274744    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.274781    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.277692    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.283935    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.316370    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.318586    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.321118    2578 log.go:25] "Validated CRI v1 image API"
Jul 01 19:22:15.581532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.322897    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 19:22:15.582128 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.325604    2578 fs.go:135] Filesystem UUIDs: map[13806640-3382-43ae-95fc-26b3922a9834:/dev/nvme0n1p4 36a0a5d7-52ef-4569-a316-ea5d440863c1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 01 19:22:15.582128 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.325619    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.331713    2578 manager.go:216] Machine: {Timestamp:2026-07-01 19:22:15.330280485 +0000 UTC m=+0.431168734 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096078 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce04db0138b0875b593126b49690e SystemUUID:ec2ce04d-b013-8b08-75b5-93126b49690e BootID:58653c6a-4c6f-49cd-883a-7c7ac47fecbe Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:9e:90:13:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:9e:90:13:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:34:72:38:57:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.331817    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.331931    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.335419    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.335446    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-138.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.336283    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.336294    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.336301    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.336315    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.337559    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342233    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342342    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342370    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342379    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342393    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342402    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342410    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 19:22:15.773489 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342904    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.342928    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.345461    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.349214    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.349269    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350083    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350098    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350105    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350111    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350120    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350131    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350185    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350196    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350208    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350218    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350272    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.350286    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.351231    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.351255    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.355621    2578 server.go:1268] "Started kubelet"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.355755    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.355783    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.355797    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-138.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.355842    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.355880    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.355878    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.357807    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.357850    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.362788    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.364690    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-138.ec2.internal.18be40c931ee1f4d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-138.ec2.internal,UID:ip-10-0-131-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-138.ec2.internal,},FirstTimestamp:2026-07-01 19:22:15.355588429 +0000 UTC m=+0.456476681,LastTimestamp:2026-07-01 19:22:15.355588429 +0000 UTC m=+0.456476681,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-138.ec2.internal,}"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.366773    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4lthh"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.368083    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.368106    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.368342    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.368875    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.368920    2578 factory.go:55] Registering systemd factory
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.368937    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369119    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369133    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369227    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369239    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 19:22:15.774309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369230    2578 factory.go:153] Registering CRI-O factory
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369260    2578 factory.go:223] Registration of the crio container factory successfully
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369309    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369333    2578 factory.go:103] Registering Raw factory
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369347    2578 manager.go:1196] Started watching for new ooms in manager
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.369738    2578 manager.go:349] Starting recovery of all containers
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.373788    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4lthh"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.378489    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.378592    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-138.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.384333    2578 manager.go:354] Recovery completed
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.393100    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395146    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395195    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395227    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395776    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395786    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395800    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.395805    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.396732    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.396761    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.396775    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.397975    2578 policy_none.go:49] "None policy: Start"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.397987    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.397997    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.400033    2578 policy_none.go:47] "Start"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.399991    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-138.ec2.internal.18be40c9344a3533  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-138.ec2.internal,UID:ip-10-0-131-138.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-138.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-138.ec2.internal,},FirstTimestamp:2026-07-01 19:22:15.395177779 +0000 UTC m=+0.496066030,LastTimestamp:2026-07-01 19:22:15.395177779 +0000 UTC m=+0.496066030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-138.ec2.internal,}"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.437475    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.438668    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.438707    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.438726    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.438761    2578 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.440934    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442265    2578 manager.go:337] "Starting Device Plugin manager"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.442291    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442302    2578 server.go:87] "Starting device plugin registration server"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442578    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442596    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442746    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442843    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.442856    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 19:22:15.775500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.443249    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.443299    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.540034    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"]
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.540115    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.542417    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.542447    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.542463    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.543478    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.544235    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.544264    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.544277    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.544304    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.545381    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.545772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.545808    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.546097    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.546119    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.546128    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.546427    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.546454    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.546466    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.548381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.548403    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.549101    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.549127    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.549138    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.556551    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.556567    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-138.ec2.internal\": node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.570941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831c2e8654fbcda304bc2de3e18ce867-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal\" (UID: \"831c2e8654fbcda304bc2de3e18ce867\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.570963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2846b3c69993af8a8bea6ca1985de70-config\") pod \"kube-apiserver-proxy-ip-10-0-131-138.ec2.internal\" (UID: \"c2846b3c69993af8a8bea6ca1985de70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.570980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/831c2e8654fbcda304bc2de3e18ce867-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal\" (UID: \"831c2e8654fbcda304bc2de3e18ce867\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.572251    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-138.ec2.internal\" not found" node="ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.575814    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-138.ec2.internal\" not found" node="ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.776878 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.622574    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.671509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2846b3c69993af8a8bea6ca1985de70-config\") pod \"kube-apiserver-proxy-ip-10-0-131-138.ec2.internal\" (UID: \"c2846b3c69993af8a8bea6ca1985de70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.671536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/831c2e8654fbcda304bc2de3e18ce867-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal\" (UID: \"831c2e8654fbcda304bc2de3e18ce867\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.671555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831c2e8654fbcda304bc2de3e18ce867-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal\" (UID: \"831c2e8654fbcda304bc2de3e18ce867\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.671606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c2846b3c69993af8a8bea6ca1985de70-config\") pod \"kube-apiserver-proxy-ip-10-0-131-138.ec2.internal\" (UID: \"c2846b3c69993af8a8bea6ca1985de70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.671683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/831c2e8654fbcda304bc2de3e18ce867-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal\" (UID: \"831c2e8654fbcda304bc2de3e18ce867\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.671712    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/831c2e8654fbcda304bc2de3e18ce867-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal\" (UID: \"831c2e8654fbcda304bc2de3e18ce867\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.777853 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.722774    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:15.987279 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.823784    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:15.987279 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.877142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.987279 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:15.880195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:15.987279 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:15.924227    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.077879 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.024807    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.164947 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.125455    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.258453 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.225898    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.275511 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.274094    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 19:22:16.275511 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.274263    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 19:22:16.355297 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.326228    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.461271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.388672    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 19:17:15 +0000 UTC" deadline="2027-12-19 18:07:09.258796487 +0000 UTC"
Jul 01 19:22:16.461271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.388708    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12862h44m52.870092833s"
Jul 01 19:22:16.461271 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.426782    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.587671 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.516111    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:22:16.587671 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.520869    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:22:16.587671 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.520900    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:22:16.587671 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.520910    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:22:16.587671 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.526842    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.587671 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.565994    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:16.703068 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.610164    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:16.703068 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.627132    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.794736 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.727342    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.894334 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:16.827764    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-138.ec2.internal\" not found"
Jul 01 19:22:16.985314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.925292    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:16.985314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.969134    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:16.985314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.982106    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:22:16.985314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.983174    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal"
Jul 01 19:22:17.074385 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:16.989762    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:22:17.132205 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.124092    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831c2e8654fbcda304bc2de3e18ce867.slice/crio-38b7b1291c24c48ceb699d3f35c2e1992e7e00065a5dae398c2bdc7df58e90e2 WatchSource:0}: Error finding container 38b7b1291c24c48ceb699d3f35c2e1992e7e00065a5dae398c2bdc7df58e90e2: Status 404 returned error can't find the container with id 38b7b1291c24c48ceb699d3f35c2e1992e7e00065a5dae398c2bdc7df58e90e2
Jul 01 19:22:17.135853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.135835    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:22:17.343850 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.343783    2578 apiserver.go:52] "Watching apiserver"
Jul 01 19:22:17.350433 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.350409    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 19:22:17.350801 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.350773    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-h65d5","openshift-multus/multus-frnld","openshift-multus/network-metrics-daemon-j9pkn","openshift-network-operator/iptables-alerter-krt2p","openshift-ovn-kubernetes/ovnkube-node-p568j","kube-system/konnectivity-agent-zgf6j","kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qp6rl","openshift-dns/node-resolver-n42f2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal","openshift-multus/multus-additional-cni-plugins-sgpsw","openshift-network-diagnostics/network-check-target-znhg2","kube-system/global-pull-secret-syncer-pjh5j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"]
Jul 01 19:22:17.354269 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.354252    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.354353 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.354315    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:17.356138 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.356123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.358175 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.358117    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.358175 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.358173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.358360 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.358222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 19:22:17.358360 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.358243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pfddm\""
Jul 01 19:22:17.358360 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.358260    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:17.358360 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.358318    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 19:22:17.358513 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.358490    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.360229 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.360215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.362047 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.362016    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 19:22:17.362047 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.362029    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.362693 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.362173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xqnst\""
Jul 01 19:22:17.362693 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.362226    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.362693 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.362440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.364419 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.364399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.364488 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.364472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.364764 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.364749    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 19:22:17.364855 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.364777    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 19:22:17.364855 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.364852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ck9fk\""
Jul 01 19:22:17.365024 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.365012    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 19:22:17.365951 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.365936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 19:22:17.365951 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.365946    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.366155 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.366142    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 19:22:17.366716 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.366698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.366832 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.366817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wsv4j\""
Jul 01 19:22:17.367016 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.366988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 19:22:17.368473 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.368460    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 19:22:17.368892 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.368873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.369020 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.369003    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pf5f9\""
Jul 01 19:22:17.369232 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.369218    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 19:22:17.369453 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.369438    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.369829 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.369814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.371558 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.371536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.372002 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.371981    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.372085 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.372014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4svtt\""
Jul 01 19:22:17.372265 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.372240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.374064 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.374046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.376144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.376129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.376238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.376222    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.376553 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.376530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.376657 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.376611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2kkkd\""
Jul 01 19:22:17.376804 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.376788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 19:22:17.376944 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.376929    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 19:22:17.377227 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.377206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mdbs7\""
Jul 01 19:22:17.377334 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.377321    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 19:22:17.378249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.378231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:17.378330 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.378286    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 19:22:17.378330 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.378296    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:17.378330 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.378318    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 19:22:17.378473 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.378336    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qv8kr\""
Jul 01 19:22:17.379962 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.379946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57fde416-4c2f-450b-9be6-46ab9b93e5fa-tmp-dir\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.380034 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.379972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-dbus\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.380034 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.379992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-kubelet\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380034 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-log-socket\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380176 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.380176 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-systemd\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380176 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.380176 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-cni-bin\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/755375e2-ca71-43d3-8845-2a84b281b425-multus-daemon-config\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-slash\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-var-lib-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-hostroot\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380309 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-run-netns\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysctl-d\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-var-lib-kubelet\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-kubelet-config\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-cni-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-k8s-cni-cncf-io\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-env-overrides\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7hjz\" (UniqueName: \"kubernetes.io/projected/652e707d-3bc9-4c49-a572-bfe0be67203d-kube-api-access-l7hjz\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-kubernetes\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-lib-modules\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-systemd\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f59f3d7-f870-44f1-88a1-a7c1de435e6a-konnectivity-ca\") pod \"konnectivity-agent-zgf6j\" (UID: \"7f59f3d7-f870-44f1-88a1-a7c1de435e6a\") " pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysconfig\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-multus-certs\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-cni-bin\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovnkube-script-lib\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-serviceca\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfnx\" (UniqueName: \"kubernetes.io/projected/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-kube-api-access-qlfnx\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-cnibin\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-os-release\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-systemd-units\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysctl-conf\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-run\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-kubelet\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovn-node-metrics-cert\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.380899 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6qs\" (UniqueName: \"kubernetes.io/projected/4c7bb6dc-bce7-4609-b96a-140c11564891-kube-api-access-pg6qs\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-cni-netd\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380932    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7bb6dc-bce7-4609-b96a-140c11564891-tmp\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-iptables-alerter-script\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.380991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-host-slash\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovnkube-config\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-host\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-sys\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/755375e2-ca71-43d3-8845-2a84b281b425-cni-binary-copy\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-conf-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f59f3d7-f870-44f1-88a1-a7c1de435e6a-agent-certs\") pod \"konnectivity-agent-zgf6j\" (UID: \"7f59f3d7-f870-44f1-88a1-a7c1de435e6a\") " pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-modprobe-d\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-host\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krq2c\" (UniqueName: \"kubernetes.io/projected/57fde416-4c2f-450b-9be6-46ab9b93e5fa-kube-api-access-krq2c\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klkf9\" (UniqueName: \"kubernetes.io/projected/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-kube-api-access-klkf9\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.381551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-etc-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381338    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-socket-dir-parent\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-node-log\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpjqp\" (UniqueName: \"kubernetes.io/projected/af186ab4-8d4a-4bf2-af43-0082dc8280e5-kube-api-access-hpjqp\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-system-cni-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-netns\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-cni-multus\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-etc-kubernetes\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pzf\" (UniqueName: \"kubernetes.io/projected/755375e2-ca71-43d3-8845-2a84b281b425-kube-api-access-r6pzf\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-ovn\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-tuned\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.382065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.381582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57fde416-4c2f-450b-9be6-46ab9b93e5fa-hosts-file\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.399501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.399485    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bjrp6"
Jul 01 19:22:17.405997 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.405982    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bjrp6"
Jul 01 19:22:17.443665 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.443599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal" event={"ID":"831c2e8654fbcda304bc2de3e18ce867","Type":"ContainerStarted","Data":"38b7b1291c24c48ceb699d3f35c2e1992e7e00065a5dae398c2bdc7df58e90e2"}
Jul 01 19:22:17.444471 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.444446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal" event={"ID":"c2846b3c69993af8a8bea6ca1985de70","Type":"ContainerStarted","Data":"aa4babbd36bdf398a55ea6c8b51b5beae48a4a1aa2a91d62d2d836dc71e6f58e"}
Jul 01 19:22:17.470138 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.470110    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 19:22:17.481777 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/755375e2-ca71-43d3-8845-2a84b281b425-multus-daemon-config\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-cnibin\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-slash\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-var-lib-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-hostroot\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-hostroot\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481920    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-slash\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.481927 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-os-release\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-var-lib-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.481986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-run-netns\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysctl-d\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-var-lib-kubelet\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-run-netns\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482118    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-kubelet-config\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-kubelet-config\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysctl-d\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-cni-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-k8s-cni-cncf-io\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-var-lib-kubelet\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-socket-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.482213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482211    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-cni-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-env-overrides\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7hjz\" (UniqueName: \"kubernetes.io/projected/652e707d-3bc9-4c49-a572-bfe0be67203d-kube-api-access-l7hjz\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482265    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/755375e2-ca71-43d3-8845-2a84b281b425-multus-daemon-config\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482264    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-k8s-cni-cncf-io\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-kubernetes\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-kubernetes\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-lib-modules\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-device-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-systemd\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482481    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-lib-modules\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f59f3d7-f870-44f1-88a1-a7c1de435e6a-konnectivity-ca\") pod \"konnectivity-agent-zgf6j\" (UID: \"7f59f3d7-f870-44f1-88a1-a7c1de435e6a\") " pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysconfig\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482538    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-systemd\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-multus-certs\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.482771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-sys-fs\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482591    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysconfig\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklx8\" (UniqueName: \"kubernetes.io/projected/7b613c19-586f-46d4-a496-bbe115025381-kube-api-access-fklx8\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482643    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-multus-certs\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-cni-bin\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovnkube-script-lib\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-serviceca\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482732    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-run-ovn-kubernetes\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfnx\" (UniqueName: \"kubernetes.io/projected/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-kube-api-access-qlfnx\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482772    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-cni-bin\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-cnibin\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482781    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-env-overrides\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-os-release\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-systemd-units\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysctl-conf\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-run\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.483528 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-kubelet\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjn2w\" (UniqueName: \"kubernetes.io/projected/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-kube-api-access-bjn2w\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovn-node-metrics-cert\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6qs\" (UniqueName: \"kubernetes.io/projected/4c7bb6dc-bce7-4609-b96a-140c11564891-kube-api-access-pg6qs\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f59f3d7-f870-44f1-88a1-a7c1de435e6a-konnectivity-ca\") pod \"konnectivity-agent-zgf6j\" (UID: \"7f59f3d7-f870-44f1-88a1-a7c1de435e6a\") " pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.482985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-cni-netd\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483023    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-run\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7bb6dc-bce7-4609-b96a-140c11564891-tmp\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-kubelet\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-iptables-alerter-script\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-cnibin\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-host-slash\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483140    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-serviceca\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovnkube-config\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-host\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.484256 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483222    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovnkube-script-lib\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-sys\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/755375e2-ca71-43d3-8845-2a84b281b425-cni-binary-copy\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483269    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-host\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-conf-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483310    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-sys\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f59f3d7-f870-44f1-88a1-a7c1de435e6a-agent-certs\") pod \"konnectivity-agent-zgf6j\" (UID: \"7f59f3d7-f870-44f1-88a1-a7c1de435e6a\") " pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-cni-netd\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483390    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-modprobe-d\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-host\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krq2c\" (UniqueName: \"kubernetes.io/projected/57fde416-4c2f-450b-9be6-46ab9b93e5fa-kube-api-access-krq2c\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klkf9\" (UniqueName: \"kubernetes.io/projected/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-kube-api-access-klkf9\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-etc-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483529    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovnkube-config\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485000 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483563    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-socket-dir-parent\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-socket-dir-parent\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-registration-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483613    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-os-release\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-iptables-alerter-script\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-node-log\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483676    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-systemd-units\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483697    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-multus-conf-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483716    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-node-log\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-etc-openvswitch\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483762    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/755375e2-ca71-43d3-8845-2a84b281b425-cni-binary-copy\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-modprobe-d\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483804    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-host-slash\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483886    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-host\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.483908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-sysctl-conf\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.485794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpjqp\" (UniqueName: \"kubernetes.io/projected/af186ab4-8d4a-4bf2-af43-0082dc8280e5-kube-api-access-hpjqp\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-system-cni-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-netns\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-cni-multus\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-etc-kubernetes\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484200    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-system-cni-dir\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pzf\" (UniqueName: \"kubernetes.io/projected/755375e2-ca71-43d3-8845-2a84b281b425-kube-api-access-r6pzf\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484230    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-run-netns\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484240    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-cni-multus\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-ovn\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-tuned\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57fde416-4c2f-450b-9be6-46ab9b93e5fa-hosts-file\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-etc-kubernetes\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57fde416-4c2f-450b-9be6-46ab9b93e5fa-tmp-dir\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-run-ovn\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57fde416-4c2f-450b-9be6-46ab9b93e5fa-hosts-file\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-dbus\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.484949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.486477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-dbus\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485447    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57fde416-4c2f-450b-9be6-46ab9b93e5fa-tmp-dir\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-system-cni-dir\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-kubelet\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-log-socket\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-systemd\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-host-kubelet\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-cni-bin\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.485766    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/755375e2-ca71-43d3-8845-2a84b281b425-host-var-lib-cni-bin\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.485845    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:17.985815357 +0000 UTC m=+3.086703610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.485888    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-systemd\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.485949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:17.985935522 +0000 UTC m=+3.086823771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.485983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af186ab4-8d4a-4bf2-af43-0082dc8280e5-log-socket\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.487228 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.486819    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c7bb6dc-bce7-4609-b96a-140c11564891-tmp\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.487988 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.486855    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c7bb6dc-bce7-4609-b96a-140c11564891-etc-tuned\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.487988 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.487061    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f59f3d7-f870-44f1-88a1-a7c1de435e6a-agent-certs\") pod \"konnectivity-agent-zgf6j\" (UID: \"7f59f3d7-f870-44f1-88a1-a7c1de435e6a\") " pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.487988 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.487504    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af186ab4-8d4a-4bf2-af43-0082dc8280e5-ovn-node-metrics-cert\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.494982 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.494962    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pzf\" (UniqueName: \"kubernetes.io/projected/755375e2-ca71-43d3-8845-2a84b281b425-kube-api-access-r6pzf\") pod \"multus-frnld\" (UID: \"755375e2-ca71-43d3-8845-2a84b281b425\") " pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.495086 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.495074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7hjz\" (UniqueName: \"kubernetes.io/projected/652e707d-3bc9-4c49-a572-bfe0be67203d-kube-api-access-l7hjz\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.495220 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.495200    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klkf9\" (UniqueName: \"kubernetes.io/projected/a5f0749b-4d82-4b63-aeb1-4beedb1bea7a-kube-api-access-klkf9\") pod \"iptables-alerter-krt2p\" (UID: \"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a\") " pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.495288 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.495264    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krq2c\" (UniqueName: \"kubernetes.io/projected/57fde416-4c2f-450b-9be6-46ab9b93e5fa-kube-api-access-krq2c\") pod \"node-resolver-n42f2\" (UID: \"57fde416-4c2f-450b-9be6-46ab9b93e5fa\") " pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.496109 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.496059    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfnx\" (UniqueName: \"kubernetes.io/projected/c4b86413-a00d-4b8c-bba9-0b094c5dacdf-kube-api-access-qlfnx\") pod \"node-ca-h65d5\" (UID: \"c4b86413-a00d-4b8c-bba9-0b094c5dacdf\") " pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.496109 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.496077    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6qs\" (UniqueName: \"kubernetes.io/projected/4c7bb6dc-bce7-4609-b96a-140c11564891-kube-api-access-pg6qs\") pod \"tuned-qp6rl\" (UID: \"4c7bb6dc-bce7-4609-b96a-140c11564891\") " pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.496339 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.496321    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpjqp\" (UniqueName: \"kubernetes.io/projected/af186ab4-8d4a-4bf2-af43-0082dc8280e5-kube-api-access-hpjqp\") pod \"ovnkube-node-p568j\" (UID: \"af186ab4-8d4a-4bf2-af43-0082dc8280e5\") " pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.586873 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.586838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.586884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-system-cni-dir\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.586940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-cnibin\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.586964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-os-release\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-socket-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-device-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-sys-fs\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklx8\" (UniqueName: \"kubernetes.io/projected/7b613c19-586f-46d4-a496-bbe115025381-kube-api-access-fklx8\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjn2w\" (UniqueName: \"kubernetes.io/projected/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-kube-api-access-bjn2w\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587281 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-registration-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587770 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587770 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-sys-fs\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587770 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.587770 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-system-cni-dir\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587770 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587581    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-cnibin\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.587770 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587654    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-os-release\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.588039 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587782    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b613c19-586f-46d4-a496-bbe115025381-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.588039 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587863    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-socket-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.588039 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587911    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-device-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.588039 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.587916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.588250 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.588036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-registration-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.588250 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.588043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.588339 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.588322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.588437 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.588416    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b613c19-586f-46d4-a496-bbe115025381-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.592880 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.592855    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:17.592880 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.592883    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:17.593054 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.592896    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:17.593054 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.592960    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:18.092942982 +0000 UTC m=+3.193831232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:17.596112 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.596053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklx8\" (UniqueName: \"kubernetes.io/projected/7b613c19-586f-46d4-a496-bbe115025381-kube-api-access-fklx8\") pod \"multus-additional-cni-plugins-sgpsw\" (UID: \"7b613c19-586f-46d4-a496-bbe115025381\") " pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.598029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.598007    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjn2w\" (UniqueName: \"kubernetes.io/projected/09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585-kube-api-access-bjn2w\") pod \"aws-ebs-csi-driver-node-hkxkw\" (UID: \"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.664834 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.664800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-frnld"
Jul 01 19:22:17.672212 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.672120    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-krt2p"
Jul 01 19:22:17.680409 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.680259    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:17.683743 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.682838    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f0749b_4d82_4b63_aeb1_4beedb1bea7a.slice/crio-2381e876a62a87f3125a5cfb6e301eb2e532b8e3608e578c422a8ec46616bec1 WatchSource:0}: Error finding container 2381e876a62a87f3125a5cfb6e301eb2e532b8e3608e578c422a8ec46616bec1: Status 404 returned error can't find the container with id 2381e876a62a87f3125a5cfb6e301eb2e532b8e3608e578c422a8ec46616bec1
Jul 01 19:22:17.683743 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.683527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:17.688980 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.688931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h65d5"
Jul 01 19:22:17.694212 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.693826    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qp6rl"
Jul 01 19:22:17.694212 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.693977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f59f3d7_f870_44f1_88a1_a7c1de435e6a.slice/crio-acc199f13d3b2472ec926e655915af85597ca9c385fc9b3c541e8753aee25fcc WatchSource:0}: Error finding container acc199f13d3b2472ec926e655915af85597ca9c385fc9b3c541e8753aee25fcc: Status 404 returned error can't find the container with id acc199f13d3b2472ec926e655915af85597ca9c385fc9b3c541e8753aee25fcc
Jul 01 19:22:17.699274 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.699122    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n42f2"
Jul 01 19:22:17.700116 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.700099    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b86413_a00d_4b8c_bba9_0b094c5dacdf.slice/crio-76a209d7feeff7ca2c3f3b2714cf1bbcc5a5638c9dc37c0590ded0973063d5d3 WatchSource:0}: Error finding container 76a209d7feeff7ca2c3f3b2714cf1bbcc5a5638c9dc37c0590ded0973063d5d3: Status 404 returned error can't find the container with id 76a209d7feeff7ca2c3f3b2714cf1bbcc5a5638c9dc37c0590ded0973063d5d3
Jul 01 19:22:17.703936 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.703808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw"
Jul 01 19:22:17.708073 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.708048    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7bb6dc_bce7_4609_b96a_140c11564891.slice/crio-860094cfb135ba33264c230f32725017eabbc1313f52c7d6ce13fc3e4daa08ec WatchSource:0}: Error finding container 860094cfb135ba33264c230f32725017eabbc1313f52c7d6ce13fc3e4daa08ec: Status 404 returned error can't find the container with id 860094cfb135ba33264c230f32725017eabbc1313f52c7d6ce13fc3e4daa08ec
Jul 01 19:22:17.708186 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.708172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgpsw"
Jul 01 19:22:17.718852 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.718789    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fde416_4c2f_450b_9be6_46ab9b93e5fa.slice/crio-b962ab66d476fec1c2ae2ee12a38362cf7c9ad73c2db0ecd11e11e6a31989f89 WatchSource:0}: Error finding container b962ab66d476fec1c2ae2ee12a38362cf7c9ad73c2db0ecd11e11e6a31989f89: Status 404 returned error can't find the container with id b962ab66d476fec1c2ae2ee12a38362cf7c9ad73c2db0ecd11e11e6a31989f89
Jul 01 19:22:17.727394 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.727315    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b613c19_586f_46d4_a496_bbe115025381.slice/crio-ed0aeccd930bd59c05c19e47b38b44219d1d7aa9bcb93a8c47091defabd0f745 WatchSource:0}: Error finding container ed0aeccd930bd59c05c19e47b38b44219d1d7aa9bcb93a8c47091defabd0f745: Status 404 returned error can't find the container with id ed0aeccd930bd59c05c19e47b38b44219d1d7aa9bcb93a8c47091defabd0f745
Jul 01 19:22:17.729778 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:17.729759    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f91aeb_6e9b_47ce_a2b9_2f1e02ce7585.slice/crio-5c71f9285acf6e76edf3a8d02982da3331839baf80813f7c9b2ce750fd6859ad WatchSource:0}: Error finding container 5c71f9285acf6e76edf3a8d02982da3331839baf80813f7c9b2ce750fd6859ad: Status 404 returned error can't find the container with id 5c71f9285acf6e76edf3a8d02982da3331839baf80813f7c9b2ce750fd6859ad
Jul 01 19:22:17.766920 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.766859    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:22:17.990285 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.990159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:17.990285 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:17.990206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:17.990487 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.990344    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:17.990487 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.990428    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:18.990406187 +0000 UTC m=+4.091294446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:17.990487 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.990347    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:17.990622 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:17.990524    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:18.990491869 +0000 UTC m=+4.091380119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:18.192350 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.192317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:18.192535 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:18.192514    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:18.192577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:18.192546    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:18.192577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:18.192559    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:18.192646 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:18.192623    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:19.192603011 +0000 UTC m=+4.293491256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:18.409550 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.409404    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:17:17 +0000 UTC" deadline="2028-03-19 11:12:04.008662072 +0000 UTC"
Jul 01 19:22:18.409550 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.409445    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15039h49m45.599222336s"
Jul 01 19:22:18.447330 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.447213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerStarted","Data":"ed0aeccd930bd59c05c19e47b38b44219d1d7aa9bcb93a8c47091defabd0f745"}
Jul 01 19:22:18.449595 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.449518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n42f2" event={"ID":"57fde416-4c2f-450b-9be6-46ab9b93e5fa","Type":"ContainerStarted","Data":"b962ab66d476fec1c2ae2ee12a38362cf7c9ad73c2db0ecd11e11e6a31989f89"}
Jul 01 19:22:18.453543 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.452545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qp6rl" event={"ID":"4c7bb6dc-bce7-4609-b96a-140c11564891","Type":"ContainerStarted","Data":"860094cfb135ba33264c230f32725017eabbc1313f52c7d6ce13fc3e4daa08ec"}
Jul 01 19:22:18.455017 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.454989    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h65d5" event={"ID":"c4b86413-a00d-4b8c-bba9-0b094c5dacdf","Type":"ContainerStarted","Data":"76a209d7feeff7ca2c3f3b2714cf1bbcc5a5638c9dc37c0590ded0973063d5d3"}
Jul 01 19:22:18.456738 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.456704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krt2p" event={"ID":"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a","Type":"ContainerStarted","Data":"2381e876a62a87f3125a5cfb6e301eb2e532b8e3608e578c422a8ec46616bec1"}
Jul 01 19:22:18.458093 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.458042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-frnld" event={"ID":"755375e2-ca71-43d3-8845-2a84b281b425","Type":"ContainerStarted","Data":"cbebe561ce10fa2c054721958e2ace091081ee10ddc958498ec6e0bd3d4c0d62"}
Jul 01 19:22:18.459456 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.459438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw" event={"ID":"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585","Type":"ContainerStarted","Data":"5c71f9285acf6e76edf3a8d02982da3331839baf80813f7c9b2ce750fd6859ad"}
Jul 01 19:22:18.460768 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.460741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zgf6j" event={"ID":"7f59f3d7-f870-44f1-88a1-a7c1de435e6a","Type":"ContainerStarted","Data":"acc199f13d3b2472ec926e655915af85597ca9c385fc9b3c541e8753aee25fcc"}
Jul 01 19:22:18.462395 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.462314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"258879d34fcdf766e196945cb379d70ce8dc42e44715ce2cd77181ca72e97edc"}
Jul 01 19:22:18.999771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.999728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:18.999977 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:18.999783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:18.999977 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:18.999968    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:19.000081 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.000030    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:21.000011798 +0000 UTC m=+6.100900064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:19.000457 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.000433    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:19.000651 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.000623    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:21.000574173 +0000 UTC m=+6.101462438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:19.201836 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:19.201798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:19.202014 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.201977    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:19.202105 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.202026    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:19.202105 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.202039    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:19.202105 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.202096    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:21.202078942 +0000 UTC m=+6.302967181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:19.410844 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:19.410728    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:17:17 +0000 UTC" deadline="2027-12-22 09:16:08.014623061 +0000 UTC"
Jul 01 19:22:19.410844 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:19.410796    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12925h53m48.603832102s"
Jul 01 19:22:19.442600 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:19.441732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:19.442600 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.441863    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:19.442600 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:19.442255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:19.442600 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.442352    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:19.442600 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:19.442407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:19.442600 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:19.442474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:21.018541 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:21.017780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:21.018541 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:21.017831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:21.018541 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.018015    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:21.018541 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.018075    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:25.018058534 +0000 UTC m=+10.118946775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:21.018541 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.018464    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:21.018541 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.018507    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:25.018493008 +0000 UTC m=+10.119381247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:21.220275 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:21.219582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:21.220275 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.219776    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:21.220275 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.219797    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:21.220275 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.219811    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:21.220275 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.219867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:25.219849135 +0000 UTC m=+10.320737374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:21.440296 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:21.439735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:21.440296 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:21.439757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:21.440296 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.439879    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:21.440296 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.439969    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:21.440296 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:21.440028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:21.440296 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:21.440105    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:23.442598 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:23.442573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:23.443012 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:23.442700    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:23.443012 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:23.442982    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:23.443094 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:23.443038    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:23.443094 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:23.443075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:23.443152 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:23.443117    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:25.049657 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:25.049546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:25.049657 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:25.049592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:25.050195 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.049727    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:25.050195 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.049800    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:33.049780457 +0000 UTC m=+18.150668701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:25.050195 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.049861    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:25.050195 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.049935    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:33.049908382 +0000 UTC m=+18.150796634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:25.251177 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:25.251140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:25.251447 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.251334    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:25.251447 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.251361    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:25.251447 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.251373    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:25.251447 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.251430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:33.251412027 +0000 UTC m=+18.352300286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:25.441315 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:25.440461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:25.441315 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.440617    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:25.441315 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:25.441005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:25.441315 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.441116    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:25.441315 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:25.441168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:25.441315 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:25.441234    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:27.439152 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:27.439110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:27.439152 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:27.439142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:27.439692 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:27.439255    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:27.439692 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:27.439354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:27.439692 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:27.439437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:27.439692 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:27.439513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:29.439901 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:29.439869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:29.440310 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:29.439880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:29.440310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:29.440000    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:29.440310 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:29.439887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:29.440310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:29.440085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:29.440310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:29.440152    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:31.439993 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:31.439961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:31.440457 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:31.439961    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:31.440457 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:31.440083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:31.440457 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:31.439977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:31.440457 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:31.440160    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:31.440457 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:31.440236    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:33.108361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:33.108318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:33.108361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:33.108359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:33.108907 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.108486    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:33.108907 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.108556    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:49.108536648 +0000 UTC m=+34.209424901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:33.108907 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.108484    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:33.108907 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.108643    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:49.108614322 +0000 UTC m=+34.209502558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:33.309684 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:33.309627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:33.309852 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.309786    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:33.309852 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.309810    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:33.309852 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.309822    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:33.309996 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.309883    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:49.309865777 +0000 UTC m=+34.410754023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:33.439500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:33.439419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:33.439500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:33.439456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:33.439740 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:33.439426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:33.439740 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.439558    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:33.439740 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.439653    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:33.439919 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:33.439736    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:35.439993 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:35.439957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:35.440527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:35.440048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:35.440527 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:35.440054    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:35.440527 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:35.440186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:35.440527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:35.440209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:35.440527 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:35.440317    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:37.439782 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.439582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:37.440062 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.439621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:37.440062 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:37.439860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:37.440062 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:37.439963    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:37.440062 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.439706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:37.440184 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:37.440078    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:37.513194 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.513156    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qp6rl" event={"ID":"4c7bb6dc-bce7-4609-b96a-140c11564891","Type":"ContainerStarted","Data":"1f10dff3460c46cdea6eef5575419a71bdf9966c5fe4e015b0dad61b0f12616b"}
Jul 01 19:22:37.515648 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.515606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-frnld" event={"ID":"755375e2-ca71-43d3-8845-2a84b281b425","Type":"ContainerStarted","Data":"299fffd5cfba81cf88929eb84fb1876d4d4d81516754fea2ae60129dbe9194e5"}
Jul 01 19:22:37.517135 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.517074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal" event={"ID":"c2846b3c69993af8a8bea6ca1985de70","Type":"ContainerStarted","Data":"2cee4f513823ea46197a3f296444ae9abfbc0218d48fae1679c147edddace3bd"}
Jul 01 19:22:37.545496 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.545451    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qp6rl" podStartSLOduration=3.024229491 podStartE2EDuration="22.545437926s" totalImagesPullingTime="19.521208435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:37.545081181 +0000 UTC m=+22.645969439" watchObservedRunningTime="2026-07-01 19:22:37.545437926 +0000 UTC m=+22.646326183"
Jul 01 19:22:37.566888 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.566840    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-138.ec2.internal" podStartSLOduration=21.566826158 podStartE2EDuration="21.566826158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:37.566530143 +0000 UTC m=+22.667418401" watchObservedRunningTime="2026-07-01 19:22:37.566826158 +0000 UTC m=+22.667714416"
Jul 01 19:22:37.595758 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:37.595705    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-frnld" podStartSLOduration=3.023315363 podStartE2EDuration="22.595689688s" totalImagesPullingTime="19.572374325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:37.59555002 +0000 UTC m=+22.696438278" watchObservedRunningTime="2026-07-01 19:22:37.595689688 +0000 UTC m=+22.696577947"
Jul 01 19:22:38.520540 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.520367    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b613c19-586f-46d4-a496-bbe115025381" containerID="a1817696df39b6c618e203341f8fa5964acd5f5a33079d81a5c2cbecac5b68e7" exitCode=0
Jul 01 19:22:38.521260 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.520401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerDied","Data":"a1817696df39b6c618e203341f8fa5964acd5f5a33079d81a5c2cbecac5b68e7"}
Jul 01 19:22:38.522136 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.522107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n42f2" event={"ID":"57fde416-4c2f-450b-9be6-46ab9b93e5fa","Type":"ContainerStarted","Data":"5fe806f3e7464416cefc426be6f5dd5fc4ce2c44e5cb41444c632bf251cefa76"}
Jul 01 19:22:38.523497 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.523473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h65d5" event={"ID":"c4b86413-a00d-4b8c-bba9-0b094c5dacdf","Type":"ContainerStarted","Data":"88fa32ef545a8855b35805e3bbae2d8f418a7fe9fab0b4c71ee6793f7dac9c3b"}
Jul 01 19:22:38.524962 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.524942    2578 generic.go:356] "Generic (PLEG): container finished" podID="831c2e8654fbcda304bc2de3e18ce867" containerID="dbf921ceabd9e32b874a3c37b098fe9376cceab580499d5dfd318db45633cb2c" exitCode=0
Jul 01 19:22:38.525067 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.525001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal" event={"ID":"831c2e8654fbcda304bc2de3e18ce867","Type":"ContainerDied","Data":"dbf921ceabd9e32b874a3c37b098fe9376cceab580499d5dfd318db45633cb2c"}
Jul 01 19:22:38.528522 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.528498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw" event={"ID":"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585","Type":"ContainerStarted","Data":"fa2d20ac8d68f2975ab4d467e1c7547e4f9c01bcb50721a5122f84e96318fb10"}
Jul 01 19:22:38.530216 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.530190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zgf6j" event={"ID":"7f59f3d7-f870-44f1-88a1-a7c1de435e6a","Type":"ContainerStarted","Data":"7a253a31f8dd7c09d8d8e3071646767e877973f363bd3c26ec6f27590586e0b8"}
Jul 01 19:22:38.533216 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.533191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"d3eb8b5fbb813e077a548886a4054ab4e8d3438b729c1e6f022485ee8d24e04b"}
Jul 01 19:22:38.533330 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.533268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"10df4207c0f484ad5029187777cc38807f39e78dedd43b8eede289ec5becca9d"}
Jul 01 19:22:38.533330 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.533321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"89302362adb08ef56b240c6133aedd22b0f7fd76432251564a58be67b1b71e31"}
Jul 01 19:22:38.533446 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.533362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"41fb68281da6c5b6e0cce4ffe2bab99af0ca1b6362f7c8bb93633c954103087c"}
Jul 01 19:22:38.533446 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.533390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"fef0f42747739b058e551002d5a804769c4eb0bf624c7bff2ca440f988f9ddf3"}
Jul 01 19:22:38.533446 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.533419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"c0d3080652613999464c9c134d5d972829fca6aba2b8c6872aac3cf1f2d6d413"}
Jul 01 19:22:38.593164 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.593109    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zgf6j" podStartSLOduration=12.436908161 podStartE2EDuration="23.593093527s" totalImagesPullingTime="11.156185366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:38.577580072 +0000 UTC m=+23.678468330" watchObservedRunningTime="2026-07-01 19:22:38.593093527 +0000 UTC m=+23.693981784"
Jul 01 19:22:38.593546 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.593519    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h65d5" podStartSLOduration=8.763438996 podStartE2EDuration="23.593511234s" totalImagesPullingTime="14.830072238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:38.592931963 +0000 UTC m=+23.693820225" watchObservedRunningTime="2026-07-01 19:22:38.593511234 +0000 UTC m=+23.694399492"
Jul 01 19:22:38.611931 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:38.611888    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n42f2" podStartSLOduration=4.098926476 podStartE2EDuration="23.611872413s" totalImagesPullingTime="19.512945937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:38.611713553 +0000 UTC m=+23.712601811" watchObservedRunningTime="2026-07-01 19:22:38.611872413 +0000 UTC m=+23.712760671"
Jul 01 19:22:39.439285 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.439257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:39.439285 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.439284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:39.439812 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:39.439378    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:39.439812 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.439430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:39.439812 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:39.439540    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:39.439812 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:39.439617    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:39.540097 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.540055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-krt2p" event={"ID":"a5f0749b-4d82-4b63-aeb1-4beedb1bea7a","Type":"ContainerStarted","Data":"ae7abe17fc13777c1ac32cd8ae14d6aed580d00008465ffb7cd6a5be31cb43e8"}
Jul 01 19:22:39.542526 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.542496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal" event={"ID":"831c2e8654fbcda304bc2de3e18ce867","Type":"ContainerStarted","Data":"1fe987ac8c01731a3fea1923ef57968872d06024187d0091ad01e92b897fd912"}
Jul 01 19:22:39.557512 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.557455    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-krt2p" podStartSLOduration=5.010983891 podStartE2EDuration="24.557436549s" totalImagesPullingTime="19.546452658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:39.556665888 +0000 UTC m=+24.657554139" watchObservedRunningTime="2026-07-01 19:22:39.557436549 +0000 UTC m=+24.658324807"
Jul 01 19:22:39.846844 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:39.846663    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 19:22:40.456728 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:40.456606    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T19:22:39.846838589Z","UUID":"e414913b-f5d6-4b44-a57f-9f7f376c8e00","Handler":null,"Name":"","Endpoint":""}
Jul 01 19:22:40.458274 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:40.458253    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 19:22:40.458430 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:40.458290    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 19:22:40.547320 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:40.547287    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw" event={"ID":"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585","Type":"ContainerStarted","Data":"047853d751dd89be39fce76379e8fc3824776a460ac89468fbd197ff236af45c"}
Jul 01 19:22:41.439218 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.439189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:41.439400 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.439195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:41.439400 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:41.439304    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:41.439400 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:41.439365    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:41.439559 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.439409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:41.439559 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:41.439529    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:41.545538 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.545501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:41.546186 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.546167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:41.553525 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.553498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"98894ac08ccabdf307c68bf3d09da8ec62dbe82129a01dd8f4698ddacc812614"}
Jul 01 19:22:41.554148 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.553572    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:41.554148 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.554043    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zgf6j"
Jul 01 19:22:41.563898 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:41.563855    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-138.ec2.internal" podStartSLOduration=25.563840197 podStartE2EDuration="25.563840197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:39.575492103 +0000 UTC m=+24.676380361" watchObservedRunningTime="2026-07-01 19:22:41.563840197 +0000 UTC m=+26.664728455"
Jul 01 19:22:43.439777 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.439596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:43.440310 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.439601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:43.440310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:43.439862    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:43.440310 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.439651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:43.440310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:43.439930    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:43.440310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:43.440006    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:43.560915 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.560873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" event={"ID":"af186ab4-8d4a-4bf2-af43-0082dc8280e5","Type":"ContainerStarted","Data":"3e79a4ed4a618009fbcbaf216f8c2034e3b39168f01126ee1d61b71cfc3f5c89"}
Jul 01 19:22:43.561109 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.561084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:43.561185 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.561151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:43.562837 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.562800    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b613c19-586f-46d4-a496-bbe115025381" containerID="25311e75e4467fa770c5149e6c700fe581c26d5c2ddd25be562557c97a5ab8db" exitCode=0
Jul 01 19:22:43.562953 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.562857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerDied","Data":"25311e75e4467fa770c5149e6c700fe581c26d5c2ddd25be562557c97a5ab8db"}
Jul 01 19:22:43.565168 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.565147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw" event={"ID":"09f91aeb-6e9b-47ce-a2b9-2f1e02ce7585","Type":"ContainerStarted","Data":"01dfb5d5e1f4af8ccaf61eb2ca0fcd6cef59f954d6d010089bc49dec6a081573"}
Jul 01 19:22:43.577565 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.577540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:43.599687 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.599622    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p568j" podStartSLOduration=8.529378841 podStartE2EDuration="28.599611004s" totalImagesPullingTime="20.070232163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:43.599361174 +0000 UTC m=+28.700249432" watchObservedRunningTime="2026-07-01 19:22:43.599611004 +0000 UTC m=+28.700499261"
Jul 01 19:22:43.701377 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:43.701282    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkxkw" podStartSLOduration=4.533875332 podStartE2EDuration="28.701266547s" totalImagesPullingTime="24.167391215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:43.665701161 +0000 UTC m=+28.766589419" watchObservedRunningTime="2026-07-01 19:22:43.701266547 +0000 UTC m=+28.802154804"
Jul 01 19:22:44.567823 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:44.567787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:44.582290 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:44.582263    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:22:45.440859 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.440678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:45.441077 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.440729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:45.441077 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:45.440950    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:45.441077 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.440761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:45.441077 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:45.441022    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:45.441300 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:45.441134    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:45.445741 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.445708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pjh5j"]
Jul 01 19:22:45.447796 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.447770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-znhg2"]
Jul 01 19:22:45.451604 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.451570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j9pkn"]
Jul 01 19:22:45.571798 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.571765    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b613c19-586f-46d4-a496-bbe115025381" containerID="30c49780b765badfd920b44d3d6fed90d9356fee61872f84fc729c7c4d59e4d5" exitCode=0
Jul 01 19:22:45.572190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.571862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerDied","Data":"30c49780b765badfd920b44d3d6fed90d9356fee61872f84fc729c7c4d59e4d5"}
Jul 01 19:22:45.572190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.571953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:45.572190 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:45.572104    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:45.572460 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.572444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:45.572674 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:45.572535    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:45.572674 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:45.572548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:45.572674 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:45.572619    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:47.439006 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:47.438948    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:47.439006 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:47.438969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:47.439594 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:47.438950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:47.439594 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:47.439069    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:47.439594 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:47.439138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:47.439594 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:47.439209    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:47.578813 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:47.578778    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b613c19-586f-46d4-a496-bbe115025381" containerID="de27dceee41ddc930ee54dc6b7304268a35df443938b38ddd77ed102e49dfa5d" exitCode=0
Jul 01 19:22:47.578969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:47.578850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerDied","Data":"de27dceee41ddc930ee54dc6b7304268a35df443938b38ddd77ed102e49dfa5d"}
Jul 01 19:22:49.126621 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:49.126587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:49.126621 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:49.126624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:49.127197 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.126743    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:49.127197 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.126749    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:49.127197 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.126799    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret podName:b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:21.126783934 +0000 UTC m=+66.227672185 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret") pod "global-pull-secret-syncer-pjh5j" (UID: "b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:22:49.127197 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.126814    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:23:21.126807043 +0000 UTC m=+66.227695279 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:22:49.328571 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:49.328534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:49.328771 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.328751    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:22:49.328831 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.328776    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:22:49.328831 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.328785    2578 projected.go:196] Error preparing data for projected volume kube-api-access-cttdm for pod openshift-network-diagnostics/network-check-target-znhg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:49.328921 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.328834    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm podName:367161fb-c70f-492d-8baa-e37640ea6adc nodeName:}" failed. No retries permitted until 2026-07-01 19:23:21.328820495 +0000 UTC m=+66.429708730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cttdm" (UniqueName: "kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm") pod "network-check-target-znhg2" (UID: "367161fb-c70f-492d-8baa-e37640ea6adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:22:49.439029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:49.438999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:49.439219 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:49.438999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:49.439219 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.439135    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-znhg2" podUID="367161fb-c70f-492d-8baa-e37640ea6adc"
Jul 01 19:22:49.439219 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:49.439000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:49.439336 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.439213    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pjh5j" podUID="b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533"
Jul 01 19:22:49.439336 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:49.439296    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j9pkn" podUID="652e707d-3bc9-4c49-a572-bfe0be67203d"
Jul 01 19:22:51.169481 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.169290    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-138.ec2.internal" event="NodeReady"
Jul 01 19:22:51.169872 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.169611    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 19:22:51.239750 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.239708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-845bc7bdd6-5cfw2"]
Jul 01 19:22:51.276723 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.276691    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"]
Jul 01 19:22:51.276892 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.276818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.281378 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.281352    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 19:22:51.281647 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.281616    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 19:22:51.281969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.281946    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fc4hm\""
Jul 01 19:22:51.281969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.281966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 19:22:51.290540 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.290516    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"]
Jul 01 19:22:51.290710 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.290690    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.291669 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.291648    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 19:22:51.294084 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.294062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 19:22:51.298011 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.296672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 19:22:51.298011 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.296720    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 19:22:51.298011 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.297186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l7q5d\""
Jul 01 19:22:51.298011 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.297364    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 01 19:22:51.308687 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.308667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"]
Jul 01 19:22:51.308804 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.308790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.318674 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.318611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 19:22:51.334670 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.334625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845bc7bdd6-5cfw2"]
Jul 01 19:22:51.334797 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.334692    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"]
Jul 01 19:22:51.334797 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.334709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"]
Jul 01 19:22:51.334797 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.334732    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lvc5r"]
Jul 01 19:22:51.334797 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.334769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.337665 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.337626    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 19:22:51.337894 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.337877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 19:22:51.338321 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.338082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 19:22:51.338321 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.338149    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 19:22:51.354849 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.354820    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"]
Jul 01 19:22:51.354849 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.354858    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dtzqg"]
Jul 01 19:22:51.355029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.354912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.358272 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.358242    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 19:22:51.358397 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.358358    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 19:22:51.361003 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.360982    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-krlqs\""
Jul 01 19:22:51.361106 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.361011    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 19:22:51.372926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.372898    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lvc5r"]
Jul 01 19:22:51.373097 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.372945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dtzqg"]
Jul 01 19:22:51.373097 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.373014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.382477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.382359    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 19:22:51.382621 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.382602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 19:22:51.382744 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.382688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2b6mr\""
Jul 01 19:22:51.439682 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.439553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:22:51.439682 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.439570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:22:51.439924 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.439821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:22:51.444249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-trusted-ca\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-ca\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.444249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnrz8\" (UniqueName: \"kubernetes.io/projected/e3145f21-bf24-4c73-911d-4b0aecb353c4-kube-api-access-fnrz8\") pod \"ingress-canary-lvc5r\" (UID: \"e3145f21-bf24-4c73-911d-4b0aecb353c4\") " pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.444249 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5128d14-493b-47eb-a7c2-2c20bae93cfc-ca-trust-extracted\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnpl\" (UniqueName: \"kubernetes.io/projected/c8869dee-a770-4020-a661-a6910f7c164c-kube-api-access-lqnpl\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zxx\" (UniqueName: \"kubernetes.io/projected/94bd21a2-02a8-4572-bb24-c096e393aa8d-kube-api-access-c5zxx\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-hub\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bd21a2-02a8-4572-bb24-c096e393aa8d-tmp-dir\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-certificates\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8869dee-a770-4020-a661-a6910f7c164c-klusterlet-config\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3145f21-bf24-4c73-911d-4b0aecb353c4-cert\") pod \"ingress-canary-lvc5r\" (UID: \"e3145f21-bf24-4c73-911d-4b0aecb353c4\") " pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8265d7dd-94f7-4dd0-8e7f-7e276a12aae9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-778dfbc5d-v7hbc\" (UID: \"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.444532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-image-registry-private-configuration\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh5z7\" (UniqueName: \"kubernetes.io/projected/8265d7dd-94f7-4dd0-8e7f-7e276a12aae9-kube-api-access-nh5z7\") pod \"managed-serviceaccount-addon-agent-778dfbc5d-v7hbc\" (UID: \"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swdv\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-kube-api-access-5swdv\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8869dee-a770-4020-a661-a6910f7c164c-tmp\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57g2k\" (UniqueName: \"kubernetes.io/projected/ad89b138-a347-40b0-bb34-5c40e054b12a-kube-api-access-57g2k\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-installation-pull-secrets\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-bound-sa-token\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94bd21a2-02a8-4572-bb24-c096e393aa8d-config-volume\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.444941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.444915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89b138-a347-40b0-bb34-5c40e054b12a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.447544 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.447522    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:51.447682 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.447551    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:51.447755 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.447731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:22:51.447897 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.447857    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:22:51.448321 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.448133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hqcw7\""
Jul 01 19:22:51.448407 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.448375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tgvrs\""
Jul 01 19:22:51.545485 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89b138-a347-40b0-bb34-5c40e054b12a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.545485 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-trusted-ca\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-ca\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnrz8\" (UniqueName: \"kubernetes.io/projected/e3145f21-bf24-4c73-911d-4b0aecb353c4-kube-api-access-fnrz8\") pod \"ingress-canary-lvc5r\" (UID: \"e3145f21-bf24-4c73-911d-4b0aecb353c4\") " pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5128d14-493b-47eb-a7c2-2c20bae93cfc-ca-trust-extracted\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnpl\" (UniqueName: \"kubernetes.io/projected/c8869dee-a770-4020-a661-a6910f7c164c-kube-api-access-lqnpl\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zxx\" (UniqueName: \"kubernetes.io/projected/94bd21a2-02a8-4572-bb24-c096e393aa8d-kube-api-access-c5zxx\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-hub\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.545751 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.545691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.546222 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bd21a2-02a8-4572-bb24-c096e393aa8d-tmp-dir\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.546343 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-certificates\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.546343 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8869dee-a770-4020-a661-a6910f7c164c-klusterlet-config\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.546343 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3145f21-bf24-4c73-911d-4b0aecb353c4-cert\") pod \"ingress-canary-lvc5r\" (UID: \"e3145f21-bf24-4c73-911d-4b0aecb353c4\") " pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.546499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad89b138-a347-40b0-bb34-5c40e054b12a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.546499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8265d7dd-94f7-4dd0-8e7f-7e276a12aae9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-778dfbc5d-v7hbc\" (UID: \"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.546499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-image-registry-private-configuration\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.546499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.546499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh5z7\" (UniqueName: \"kubernetes.io/projected/8265d7dd-94f7-4dd0-8e7f-7e276a12aae9-kube-api-access-nh5z7\") pod \"managed-serviceaccount-addon-agent-778dfbc5d-v7hbc\" (UID: \"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.546499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swdv\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-kube-api-access-5swdv\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8869dee-a770-4020-a661-a6910f7c164c-tmp\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57g2k\" (UniqueName: \"kubernetes.io/projected/ad89b138-a347-40b0-bb34-5c40e054b12a-kube-api-access-57g2k\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-installation-pull-secrets\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-bound-sa-token\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.546773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94bd21a2-02a8-4572-bb24-c096e393aa8d-config-volume\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.547041 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546865    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5128d14-493b-47eb-a7c2-2c20bae93cfc-ca-trust-extracted\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.547041 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-trusted-ca\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.547862 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.547117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94bd21a2-02a8-4572-bb24-c096e393aa8d-config-volume\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.547862 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.547131    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8869dee-a770-4020-a661-a6910f7c164c-tmp\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.547862 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.546606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94bd21a2-02a8-4572-bb24-c096e393aa8d-tmp-dir\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.547862 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:51.547554    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:51.547862 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:51.547611    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:52.047591478 +0000 UTC m=+37.148479721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:51.548183 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.547987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-certificates\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.548183 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:51.548123    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:51.548183 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:51.548138    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:22:51.548325 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:51.548187    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:52.048167558 +0000 UTC m=+37.149055793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:22:51.551213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.551156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8869dee-a770-4020-a661-a6910f7c164c-klusterlet-config\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.551213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.551205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-image-registry-private-configuration\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.551394 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.551233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8265d7dd-94f7-4dd0-8e7f-7e276a12aae9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-778dfbc5d-v7hbc\" (UID: \"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.551394 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.551384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.551569 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.551535    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-ca\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.552435 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.552377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.552543 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.552504    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ad89b138-a347-40b0-bb34-5c40e054b12a-hub\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.552680 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.552655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-installation-pull-secrets\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.552896 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.552880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3145f21-bf24-4c73-911d-4b0aecb353c4-cert\") pod \"ingress-canary-lvc5r\" (UID: \"e3145f21-bf24-4c73-911d-4b0aecb353c4\") " pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.571300 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.571268    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zxx\" (UniqueName: \"kubernetes.io/projected/94bd21a2-02a8-4572-bb24-c096e393aa8d-kube-api-access-c5zxx\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:51.572510 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.572485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnpl\" (UniqueName: \"kubernetes.io/projected/c8869dee-a770-4020-a661-a6910f7c164c-kube-api-access-lqnpl\") pod \"klusterlet-addon-workmgr-699fd6bd8b-kxf2x\" (UID: \"c8869dee-a770-4020-a661-a6910f7c164c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.575189 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.575164    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnrz8\" (UniqueName: \"kubernetes.io/projected/e3145f21-bf24-4c73-911d-4b0aecb353c4-kube-api-access-fnrz8\") pod \"ingress-canary-lvc5r\" (UID: \"e3145f21-bf24-4c73-911d-4b0aecb353c4\") " pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.582398 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.582318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-bound-sa-token\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.582984 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.582964    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57g2k\" (UniqueName: \"kubernetes.io/projected/ad89b138-a347-40b0-bb34-5c40e054b12a-kube-api-access-57g2k\") pod \"cluster-proxy-proxy-agent-b6d74ddb-lz2k4\" (UID: \"ad89b138-a347-40b0-bb34-5c40e054b12a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.583661 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.583620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swdv\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-kube-api-access-5swdv\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:51.594797 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.594771    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh5z7\" (UniqueName: \"kubernetes.io/projected/8265d7dd-94f7-4dd0-8e7f-7e276a12aae9-kube-api-access-nh5z7\") pod \"managed-serviceaccount-addon-agent-778dfbc5d-v7hbc\" (UID: \"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.619973 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.619946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"
Jul 01 19:22:51.631901 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.631880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:22:51.633417 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.633389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:51.645259 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.645235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:22:51.677864 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.677825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lvc5r"
Jul 01 19:22:51.756516 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.756408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:51.885136 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.885101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"]
Jul 01 19:22:51.886217 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.886192    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc"]
Jul 01 19:22:51.886538 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:51.886484    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad89b138_a347_40b0_bb34_5c40e054b12a.slice/crio-fea36a3c951375f77bff46e2b2931cb3650c4f23981940d912fcf2693390908f WatchSource:0}: Error finding container fea36a3c951375f77bff46e2b2931cb3650c4f23981940d912fcf2693390908f: Status 404 returned error can't find the container with id fea36a3c951375f77bff46e2b2931cb3650c4f23981940d912fcf2693390908f
Jul 01 19:22:51.887889 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.887855    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"]
Jul 01 19:22:51.888699 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:51.888681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lvc5r"]
Jul 01 19:22:51.891568 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:51.891542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8869dee_a770_4020_a661_a6910f7c164c.slice/crio-99dbb5d8cc9698e6c04c5c4a56775998c072c653c4912895c1ddd9479c245cdb WatchSource:0}: Error finding container 99dbb5d8cc9698e6c04c5c4a56775998c072c653c4912895c1ddd9479c245cdb: Status 404 returned error can't find the container with id 99dbb5d8cc9698e6c04c5c4a56775998c072c653c4912895c1ddd9479c245cdb
Jul 01 19:22:51.892186 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:51.892164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3145f21_bf24_4c73_911d_4b0aecb353c4.slice/crio-bfd6a35684466c581ed6ee18e032bc315a98b1c33cbf7a2e6655d0f8aad82cdc WatchSource:0}: Error finding container bfd6a35684466c581ed6ee18e032bc315a98b1c33cbf7a2e6655d0f8aad82cdc: Status 404 returned error can't find the container with id bfd6a35684466c581ed6ee18e032bc315a98b1c33cbf7a2e6655d0f8aad82cdc
Jul 01 19:22:51.892964 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:22:51.892943    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8265d7dd_94f7_4dd0_8e7f_7e276a12aae9.slice/crio-4afcf9b19db41a62a3c912af332702075b93c834aaa0ad7444fb7c45f9e850d5 WatchSource:0}: Error finding container 4afcf9b19db41a62a3c912af332702075b93c834aaa0ad7444fb7c45f9e850d5: Status 404 returned error can't find the container with id 4afcf9b19db41a62a3c912af332702075b93c834aaa0ad7444fb7c45f9e850d5
Jul 01 19:22:52.050393 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.050183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:52.050539 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.050419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:52.050539 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:52.050325    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:52.050539 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:52.050517    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:53.05049775 +0000 UTC m=+38.151386006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:52.050664 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:52.050561    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:52.050664 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:52.050571    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:22:52.050664 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:52.050601    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:53.050594299 +0000 UTC m=+38.151482543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:22:52.594970 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.594932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" event={"ID":"ad89b138-a347-40b0-bb34-5c40e054b12a","Type":"ContainerStarted","Data":"fea36a3c951375f77bff46e2b2931cb3650c4f23981940d912fcf2693390908f"}
Jul 01 19:22:52.596229 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.596199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc" event={"ID":"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9","Type":"ContainerStarted","Data":"4afcf9b19db41a62a3c912af332702075b93c834aaa0ad7444fb7c45f9e850d5"}
Jul 01 19:22:52.599112 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.599087    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b613c19-586f-46d4-a496-bbe115025381" containerID="8f0d69eea01b29702453cffadfc963870383d3060b98d438d8603ac9f70881ab" exitCode=0
Jul 01 19:22:52.599241 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.599186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerDied","Data":"8f0d69eea01b29702453cffadfc963870383d3060b98d438d8603ac9f70881ab"}
Jul 01 19:22:52.600445 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.600421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lvc5r" event={"ID":"e3145f21-bf24-4c73-911d-4b0aecb353c4","Type":"ContainerStarted","Data":"bfd6a35684466c581ed6ee18e032bc315a98b1c33cbf7a2e6655d0f8aad82cdc"}
Jul 01 19:22:52.601616 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:52.601587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x" event={"ID":"c8869dee-a770-4020-a661-a6910f7c164c","Type":"ContainerStarted","Data":"99dbb5d8cc9698e6c04c5c4a56775998c072c653c4912895c1ddd9479c245cdb"}
Jul 01 19:22:53.020512 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.020480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.059143 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.059083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:53.059666 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.059377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:53.059666 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:53.059527    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:53.059666 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:53.059549    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:22:53.059666 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:53.059609    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:55.059589763 +0000 UTC m=+40.160478041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:22:53.059953 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:53.059698    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:53.059953 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:53.059749    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:55.059734106 +0000 UTC m=+40.160622360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:53.244863 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.244825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.390400 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.390129    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.503315 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.503251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.607963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.607815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.613682 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.613570    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b613c19-586f-46d4-a496-bbe115025381" containerID="dac73fb20a205951e0235de6e2e2e8b569eae3652bd532663dd794e1f85d1830" exitCode=0
Jul 01 19:22:53.613682 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.613657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerDied","Data":"dac73fb20a205951e0235de6e2e2e8b569eae3652bd532663dd794e1f85d1830"}
Jul 01 19:22:53.719272 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.719208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.800087 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.800037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:53.908857 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:53.908821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.344935 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:54.344869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.418617 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:54.418580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.741332 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:54.741294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.866568 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:54.866504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:54.940437 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:54.940120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:55.045183 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:55.044906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:55.081345 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:55.081310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:55.081345 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:55.081364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:55.081585 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:55.081446    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:55.081585 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:55.081533    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:55.081585 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:55.081545    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:22:55.081585 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:55.081562    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:22:59.081541694 +0000 UTC m=+44.182429940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:55.081585 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:55.081578    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:22:59.081568839 +0000 UTC m=+44.182457074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:22:56.850999 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:56.850968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:56.971253 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:56.971213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.080823 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.080786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.182798 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.182768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.267175 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.267144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.627958 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.627675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" event={"ID":"ad89b138-a347-40b0-bb34-5c40e054b12a","Type":"ContainerStarted","Data":"14c2be6e1c7556bcf1d245707458faefc6854acc5961274afcf651767540c956"}
Jul 01 19:22:57.629486 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.629458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc" event={"ID":"8265d7dd-94f7-4dd0-8e7f-7e276a12aae9","Type":"ContainerStarted","Data":"4b03a0a1fea8b8f6f511e21eb1204184e7eace0de38a29edaecc17dffb2bb5d3"}
Jul 01 19:22:57.633433 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.633392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" event={"ID":"7b613c19-586f-46d4-a496-bbe115025381","Type":"ContainerStarted","Data":"d9f2100fbd48680ec2796a463b4d66311d230b2ed4e320f8bd76802f3520b117"}
Jul 01 19:22:57.634810 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.634782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lvc5r" event={"ID":"e3145f21-bf24-4c73-911d-4b0aecb353c4","Type":"ContainerStarted","Data":"2ef8316f51c836b587fac9c17113d2ea64e05f7013cba16c7828a1f9aca1eef1"}
Jul 01 19:22:57.657932 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.657854    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-778dfbc5d-v7hbc" podStartSLOduration=28.117866034 podStartE2EDuration="33.657831681s" totalImagesPullingTime="5.539965647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:57.657180966 +0000 UTC m=+42.758069225" watchObservedRunningTime="2026-07-01 19:22:57.657831681 +0000 UTC m=+42.758719942"
Jul 01 19:22:57.723560 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.723516    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lvc5r" podStartSLOduration=1.214188327 podStartE2EDuration="6.723499619s" totalImagesPullingTime="5.509311292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:57.723311102 +0000 UTC m=+42.824199367" watchObservedRunningTime="2026-07-01 19:22:57.723499619 +0000 UTC m=+42.824387876"
Jul 01 19:22:57.723748 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.723725    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sgpsw" podStartSLOduration=12.131997959 podStartE2EDuration="42.723717391s" totalImagesPullingTime="30.591719432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:57.698896959 +0000 UTC m=+42.799785219" watchObservedRunningTime="2026-07-01 19:22:57.723717391 +0000 UTC m=+42.824605650"
Jul 01 19:22:57.916581 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.916543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:57.981884 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:57.981792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.303964 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:58.303888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.373002 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:58.372972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:58.639584 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:58.639508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" event={"ID":"ad89b138-a347-40b0-bb34-5c40e054b12a","Type":"ContainerStarted","Data":"8c66101ae21ef1d3031996c89d5b60e9f669f44227d333aea0e8f14b9ff1335b"}
Jul 01 19:22:58.639584 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:58.639546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" event={"ID":"ad89b138-a347-40b0-bb34-5c40e054b12a","Type":"ContainerStarted","Data":"a5b7fbadb1dc7587a4cae96f6c01be7f0640304beec66eebb9f96f488bac7ffe"}
Jul 01 19:22:58.671282 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:58.671232    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" podStartSLOduration=29.112798573 podStartE2EDuration="34.671215515s" totalImagesPullingTime="5.558416942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:58.669466487 +0000 UTC m=+43.770354747" watchObservedRunningTime="2026-07-01 19:22:58.671215515 +0000 UTC m=+43.772103772"
Jul 01 19:22:59.109209 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:59.109165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:59.117106 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:59.117083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:22:59.117198 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:59.117121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:22:59.117265 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:59.117248    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:59.117329 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:59.117318    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:23:07.117298513 +0000 UTC m=+52.218186763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:59.117411 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:59.117251    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:59.117411 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:59.117351    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:22:59.117411 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:22:59.117392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:23:07.117381573 +0000 UTC m=+52.218269809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:22:59.215102 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:59.215067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:59.514022 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:59.513983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:22:59.620005 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:22:59.619974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:07.178724 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:07.178686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:23:07.178724 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:07.178731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:23:07.179224 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:07.178833    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:23:07.179224 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:07.178859    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:23:07.179224 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:07.178869    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:23:07.179224 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:07.178904    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:23:23.178889135 +0000 UTC m=+68.279777371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:23:07.179224 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:07.178919    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:23:23.178913442 +0000 UTC m=+68.279801678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:23:10.672182 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:10.672153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x" event={"ID":"c8869dee-a770-4020-a661-a6910f7c164c","Type":"ContainerStarted","Data":"95a68ab24b1fe4c0c56aaf5c64f2968d79fb949c523829e4667c592f226a19ba"}
Jul 01 19:23:10.672544 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:10.672361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:23:10.673950 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:10.673930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x"
Jul 01 19:23:10.694331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:10.694277    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-699fd6bd8b-kxf2x" podStartSLOduration=28.22098688 podStartE2EDuration="46.694260847s" totalImagesPullingTime="18.473273967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:10.694215361 +0000 UTC m=+55.795103619" watchObservedRunningTime="2026-07-01 19:23:10.694260847 +0000 UTC m=+55.795149102"
Jul 01 19:23:11.023951 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:11.023855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:11.083568 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:11.083537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log" limit=1024
Jul 01 19:23:16.585213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:16.585186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p568j"
Jul 01 19:23:21.181016 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.180972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:23:21.181016 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.181016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:23:21.183497 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.183473    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:23:21.183598 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.183583    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:23:21.191604 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:21.191584    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 19:23:21.191674 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:21.191663    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs podName:652e707d-3bc9-4c49-a572-bfe0be67203d nodeName:}" failed. No retries permitted until 2026-07-01 19:24:25.191646596 +0000 UTC m=+130.292534846 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs") pod "network-metrics-daemon-j9pkn" (UID: "652e707d-3bc9-4c49-a572-bfe0be67203d") : secret "metrics-daemon-secret" not found
Jul 01 19:23:21.194716 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.194687    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533-original-pull-secret\") pod \"global-pull-secret-syncer-pjh5j\" (UID: \"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533\") " pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:23:21.382803 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.382764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:23:21.385409 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.385390    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:23:21.395794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.395774    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:23:21.407021 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.406996    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttdm\" (UniqueName: \"kubernetes.io/projected/367161fb-c70f-492d-8baa-e37640ea6adc-kube-api-access-cttdm\") pod \"network-check-target-znhg2\" (UID: \"367161fb-c70f-492d-8baa-e37640ea6adc\") " pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:23:21.458966 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.458943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pjh5j"
Jul 01 19:23:21.468926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.468907    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tgvrs\""
Jul 01 19:23:21.477157 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.477137    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:23:21.593542 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.593514    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pjh5j"]
Jul 01 19:23:21.596715 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:23:21.596690    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e1ba6d_c522_4182_b0b9_e5b3f2cbb533.slice/crio-e14ab6a1ff4cfb06277c7f232132c23e62a5b3bb6cbbedc91cea06aac6bf55c6 WatchSource:0}: Error finding container e14ab6a1ff4cfb06277c7f232132c23e62a5b3bb6cbbedc91cea06aac6bf55c6: Status 404 returned error can't find the container with id e14ab6a1ff4cfb06277c7f232132c23e62a5b3bb6cbbedc91cea06aac6bf55c6
Jul 01 19:23:21.609535 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.609507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-znhg2"]
Jul 01 19:23:21.612081 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:23:21.612053    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367161fb_c70f_492d_8baa_e37640ea6adc.slice/crio-31eb7fe8eec9596a2e0ab29ad4b20c2ec639e61c6ffe816eb828e6d8dcc5c2dc WatchSource:0}: Error finding container 31eb7fe8eec9596a2e0ab29ad4b20c2ec639e61c6ffe816eb828e6d8dcc5c2dc: Status 404 returned error can't find the container with id 31eb7fe8eec9596a2e0ab29ad4b20c2ec639e61c6ffe816eb828e6d8dcc5c2dc
Jul 01 19:23:21.705909 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.705818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-znhg2" event={"ID":"367161fb-c70f-492d-8baa-e37640ea6adc","Type":"ContainerStarted","Data":"31eb7fe8eec9596a2e0ab29ad4b20c2ec639e61c6ffe816eb828e6d8dcc5c2dc"}
Jul 01 19:23:21.706934 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:21.706903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pjh5j" event={"ID":"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533","Type":"ContainerStarted","Data":"e14ab6a1ff4cfb06277c7f232132c23e62a5b3bb6cbbedc91cea06aac6bf55c6"}
Jul 01 19:23:23.199102 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:23.199063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:23:23.199577 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:23.199115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:23:23.199577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:23.199227    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:23:23.199577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:23.199297    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.199281237 +0000 UTC m=+100.300169495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:23:23.199577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:23.199238    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:23:23.199577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:23.199322    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:23:23.199577 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:23.199359    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:23:55.199350464 +0000 UTC m=+100.300238716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:23:27.730915 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:27.730877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pjh5j" event={"ID":"b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533","Type":"ContainerStarted","Data":"8db3759ad60038480e664233491368e741395796f6d8be060fe7fa90ee247ea3"}
Jul 01 19:23:27.732239 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:27.732214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-znhg2" event={"ID":"367161fb-c70f-492d-8baa-e37640ea6adc","Type":"ContainerStarted","Data":"9830fbb1d12c65f409bec436c3f856e909fe6b213b836cfe5ae9eaf8b86cb327"}
Jul 01 19:23:27.732354 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:27.732341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:23:27.747786 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:27.747597    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pjh5j" podStartSLOduration=67.588231071 podStartE2EDuration="1m12.747580774s" totalImagesPullingTime="5.159349703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:27.747302752 +0000 UTC m=+72.848191010" watchObservedRunningTime="2026-07-01 19:23:27.747580774 +0000 UTC m=+72.848469035"
Jul 01 19:23:27.763947 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:27.763901    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-znhg2" podStartSLOduration=67.610471505 podStartE2EDuration="1m12.763887283s" totalImagesPullingTime="5.153415778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:27.763139817 +0000 UTC m=+72.864028074" watchObservedRunningTime="2026-07-01 19:23:27.763887283 +0000 UTC m=+72.864775540"
Jul 01 19:23:55.243689 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:55.243617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:23:55.243689 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:55.243695    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") pod \"image-registry-845bc7bdd6-5cfw2\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") " pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:23:55.244284 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:55.243793    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:23:55.244284 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:55.243847    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:23:55.244284 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:55.243863    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845bc7bdd6-5cfw2: secret "image-registry-tls" not found
Jul 01 19:23:55.244284 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:55.243877    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls podName:94bd21a2-02a8-4572-bb24-c096e393aa8d nodeName:}" failed. No retries permitted until 2026-07-01 19:24:59.243854507 +0000 UTC m=+164.344742756 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls") pod "dns-default-dtzqg" (UID: "94bd21a2-02a8-4572-bb24-c096e393aa8d") : secret "dns-default-metrics-tls" not found
Jul 01 19:23:55.244284 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:55.243917    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls podName:c5128d14-493b-47eb-a7c2-2c20bae93cfc nodeName:}" failed. No retries permitted until 2026-07-01 19:24:59.24390465 +0000 UTC m=+164.344792891 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls") pod "image-registry-845bc7bdd6-5cfw2" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc") : secret "image-registry-tls" not found
Jul 01 19:23:58.737821 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:58.737791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-znhg2"
Jul 01 19:23:59.480186 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.480152    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lrzs5"]
Jul 01 19:23:59.515582 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.515548    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrzs5"]
Jul 01 19:23:59.515747 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.515674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.518771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.518746    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 19:23:59.519865 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.519846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 19:23:59.523706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.523688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 19:23:59.523780 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.523706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fr8gc\""
Jul 01 19:23:59.523780 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.523706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 19:23:59.575197 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.575168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.575197 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.575204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81a32760-9a0f-4f9b-b430-be68d041bed3-data-volume\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.575438 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.575227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkpg\" (UniqueName: \"kubernetes.io/projected/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-api-access-kfkpg\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.575438 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.575248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.575438 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.575365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81a32760-9a0f-4f9b-b430-be68d041bed3-crio-socket\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676060 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81a32760-9a0f-4f9b-b430-be68d041bed3-data-volume\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkpg\" (UniqueName: \"kubernetes.io/projected/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-api-access-kfkpg\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81a32760-9a0f-4f9b-b430-be68d041bed3-crio-socket\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676335 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676232    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/81a32760-9a0f-4f9b-b430-be68d041bed3-crio-socket\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676335 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:59.676272    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:59.676335 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:59.676325    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:00.176311179 +0000 UTC m=+105.277199415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:23:59.676478 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676478 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:59.676433    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:23:59.676478 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.676448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/81a32760-9a0f-4f9b-b430-be68d041bed3-data-volume\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:23:59.676478 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:23:59.676471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:00.176462658 +0000 UTC m=+105.277350895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : configmap "kube-rbac-proxy" not found
Jul 01 19:23:59.693967 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:23:59.693940    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkpg\" (UniqueName: \"kubernetes.io/projected/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-api-access-kfkpg\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:00.179777 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:00.179742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:00.180158 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:00.179800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:00.180158 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:00.179863    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:24:00.180158 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:00.179902    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:00.180158 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:00.179932    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:01.179915753 +0000 UTC m=+106.280804006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : configmap "kube-rbac-proxy" not found
Jul 01 19:24:00.180158 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:00.179950    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:01.179939494 +0000 UTC m=+106.280827731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:01.189490 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:01.189454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:01.189917 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:01.189512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:01.189917 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:01.189577    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:24:01.189917 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:01.189628    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:01.189917 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:01.189660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.189628284 +0000 UTC m=+108.290516520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : configmap "kube-rbac-proxy" not found
Jul 01 19:24:01.189917 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:01.189689    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:03.189678164 +0000 UTC m=+108.290566399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:01.469156 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:01.469086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n42f2_57fde416-4c2f-450b-9be6-46ab9b93e5fa/dns-node-resolver/0.log"
Jul 01 19:24:02.462683 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.462656    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h65d5_c4b86413-a00d-4b8c-bba9-0b094c5dacdf/node-ca/0.log"
Jul 01 19:24:02.658731 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.658701    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"]
Jul 01 19:24:02.660886 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.660864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"
Jul 01 19:24:02.664232 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.664206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-79wm2\""
Jul 01 19:24:02.675404 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.675383    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"]
Jul 01 19:24:02.801954 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.801856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbltt\" (UniqueName: \"kubernetes.io/projected/00fea737-05aa-4586-ae48-68dc2b2e5786-kube-api-access-rbltt\") pod \"network-check-source-56499946bc-wtqpz\" (UID: \"00fea737-05aa-4586-ae48-68dc2b2e5786\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"
Jul 01 19:24:02.902479 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.902243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbltt\" (UniqueName: \"kubernetes.io/projected/00fea737-05aa-4586-ae48-68dc2b2e5786-kube-api-access-rbltt\") pod \"network-check-source-56499946bc-wtqpz\" (UID: \"00fea737-05aa-4586-ae48-68dc2b2e5786\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"
Jul 01 19:24:02.912100 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.912070    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbltt\" (UniqueName: \"kubernetes.io/projected/00fea737-05aa-4586-ae48-68dc2b2e5786-kube-api-access-rbltt\") pod \"network-check-source-56499946bc-wtqpz\" (UID: \"00fea737-05aa-4586-ae48-68dc2b2e5786\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"
Jul 01 19:24:02.970051 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:02.970020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"
Jul 01 19:24:03.068901 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.068823    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvc5r_e3145f21-bf24-4c73-911d-4b0aecb353c4/serve-healthcheck-canary/0.log"
Jul 01 19:24:03.092435 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.092404    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-wtqpz"]
Jul 01 19:24:03.095323 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:24:03.095294    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fea737_05aa_4586_ae48_68dc2b2e5786.slice/crio-86c445b3906788375772559feae941d996c57287f7941276190ecc4c3e3fb0d9 WatchSource:0}: Error finding container 86c445b3906788375772559feae941d996c57287f7941276190ecc4c3e3fb0d9: Status 404 returned error can't find the container with id 86c445b3906788375772559feae941d996c57287f7941276190ecc4c3e3fb0d9
Jul 01 19:24:03.205592 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.205558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:03.205796 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.205652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:03.205796 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:03.205744    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:24:03.205796 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:03.205758    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:03.205928 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:03.205801    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:07.205784506 +0000 UTC m=+112.306672765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : configmap "kube-rbac-proxy" not found
Jul 01 19:24:03.205928 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:03.205840    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:07.205820347 +0000 UTC m=+112.306708583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:03.837109 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.837075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz" event={"ID":"00fea737-05aa-4586-ae48-68dc2b2e5786","Type":"ContainerStarted","Data":"4d4499be939a3f6c27c3d927b97a7c247ba275b2467ad36562bd0105791c62ab"}
Jul 01 19:24:03.837500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.837128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz" event={"ID":"00fea737-05aa-4586-ae48-68dc2b2e5786","Type":"ContainerStarted","Data":"86c445b3906788375772559feae941d996c57287f7941276190ecc4c3e3fb0d9"}
Jul 01 19:24:03.861242 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:03.861178    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-wtqpz" podStartSLOduration=1.8611614250000001 podStartE2EDuration="1.861161425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:03.86037401 +0000 UTC m=+108.961262267" watchObservedRunningTime="2026-07-01 19:24:03.861161425 +0000 UTC m=+108.962049729"
Jul 01 19:24:07.234375 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:07.234336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:07.234839 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:07.234411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:07.234839 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:07.234474    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:24:07.234839 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:07.234474    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:07.234839 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:07.234531    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:15.234517359 +0000 UTC m=+120.335405596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : configmap "kube-rbac-proxy" not found
Jul 01 19:24:07.234839 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:07.234545    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:15.234539441 +0000 UTC m=+120.335427677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:15.299036 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:15.299003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:15.299443 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:15.299112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:15.299443 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:15.299172    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:15.299443 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:15.299230    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls podName:81a32760-9a0f-4f9b-b430-be68d041bed3 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:31.299216656 +0000 UTC m=+136.400104899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lrzs5" (UID: "81a32760-9a0f-4f9b-b430-be68d041bed3") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:24:15.299685 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:15.299664    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/81a32760-9a0f-4f9b-b430-be68d041bed3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:25.276914 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:25.276858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:24:25.279251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:25.279227    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/652e707d-3bc9-4c49-a572-bfe0be67203d-metrics-certs\") pod \"network-metrics-daemon-j9pkn\" (UID: \"652e707d-3bc9-4c49-a572-bfe0be67203d\") " pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:24:25.355794 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:25.355764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hqcw7\""
Jul 01 19:24:25.361975 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:25.361953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j9pkn"
Jul 01 19:24:25.494286 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:25.494245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j9pkn"]
Jul 01 19:24:25.497075 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:24:25.497049    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652e707d_3bc9_4c49_a572_bfe0be67203d.slice/crio-34ad5288f9fe7329bb71421bd2fc6df930e0f7f82739c30b1d9aaadf7e82c06a WatchSource:0}: Error finding container 34ad5288f9fe7329bb71421bd2fc6df930e0f7f82739c30b1d9aaadf7e82c06a: Status 404 returned error can't find the container with id 34ad5288f9fe7329bb71421bd2fc6df930e0f7f82739c30b1d9aaadf7e82c06a
Jul 01 19:24:25.899812 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:25.899770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j9pkn" event={"ID":"652e707d-3bc9-4c49-a572-bfe0be67203d","Type":"ContainerStarted","Data":"34ad5288f9fe7329bb71421bd2fc6df930e0f7f82739c30b1d9aaadf7e82c06a"}
Jul 01 19:24:26.904139 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:26.904104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j9pkn" event={"ID":"652e707d-3bc9-4c49-a572-bfe0be67203d","Type":"ContainerStarted","Data":"3759bb34af82d9722eb30e03e78d017a25cf1f29017a101cea50cfdfe99d77f9"}
Jul 01 19:24:26.904566 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:26.904166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j9pkn" event={"ID":"652e707d-3bc9-4c49-a572-bfe0be67203d","Type":"ContainerStarted","Data":"1611587ceadc0dae95cfc70146783faf2edf33157d22cba61e64a1db7b0cc99b"}
Jul 01 19:24:26.922196 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:26.922147    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j9pkn" podStartSLOduration=130.957451523 podStartE2EDuration="2m11.922131858s" totalImagesPullingTime="964.680335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:26.920663624 +0000 UTC m=+132.021551874" watchObservedRunningTime="2026-07-01 19:24:26.922131858 +0000 UTC m=+132.023020116"
Jul 01 19:24:31.323996 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:31.323936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:31.326206 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:31.326182    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/81a32760-9a0f-4f9b-b430-be68d041bed3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzs5\" (UID: \"81a32760-9a0f-4f9b-b430-be68d041bed3\") " pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:31.627796 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:31.627719    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fr8gc\""
Jul 01 19:24:31.636271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:31.636249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrzs5"
Jul 01 19:24:31.765729 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:31.765699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrzs5"]
Jul 01 19:24:31.769969 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:24:31.769941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a32760_9a0f_4f9b_b430_be68d041bed3.slice/crio-03480c004c7045ffe9a1a951e838498b1b71cdbccc12e89d55a05e0266988465 WatchSource:0}: Error finding container 03480c004c7045ffe9a1a951e838498b1b71cdbccc12e89d55a05e0266988465: Status 404 returned error can't find the container with id 03480c004c7045ffe9a1a951e838498b1b71cdbccc12e89d55a05e0266988465
Jul 01 19:24:31.920268 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:31.920237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzs5" event={"ID":"81a32760-9a0f-4f9b-b430-be68d041bed3","Type":"ContainerStarted","Data":"03480c004c7045ffe9a1a951e838498b1b71cdbccc12e89d55a05e0266988465"}
Jul 01 19:24:35.934552 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:35.934509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzs5" event={"ID":"81a32760-9a0f-4f9b-b430-be68d041bed3","Type":"ContainerStarted","Data":"e2b4532f48a0b96addad0ef5478236eaa5ae456ffa4a65fcbd30a82eaa2c355e"}
Jul 01 19:24:36.940440 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:36.940404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzs5" event={"ID":"81a32760-9a0f-4f9b-b430-be68d041bed3","Type":"ContainerStarted","Data":"b2a1b7886f1fc5cde77670d5a47638cfbd66fcf3935905d165a86945463f9488"}
Jul 01 19:24:38.948177 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:38.948135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzs5" event={"ID":"81a32760-9a0f-4f9b-b430-be68d041bed3","Type":"ContainerStarted","Data":"0bea9aea3fb0f2aad9bc676a3b4fe001d60d2c791da0d5ff43ac0f6575124daf"}
Jul 01 19:24:38.966775 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:38.966729    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lrzs5" podStartSLOduration=33.07571084 podStartE2EDuration="39.966715323s" totalImagesPullingTime="6.891004483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:23:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:38.965494462 +0000 UTC m=+144.066382733" watchObservedRunningTime="2026-07-01 19:24:38.966715323 +0000 UTC m=+144.067603580"
Jul 01 19:24:48.352172 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:48.352137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-845bc7bdd6-5cfw2"]
Jul 01 19:24:48.352554 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:48.352357    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2" podUID="c5128d14-493b-47eb-a7c2-2c20bae93cfc"
Jul 01 19:24:48.978043 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:48.978011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:24:48.984389 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:48.984371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:24:49.067145 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-trusted-ca\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067306 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067167    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5swdv\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-kube-api-access-5swdv\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067306 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-bound-sa-token\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067306 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5128d14-493b-47eb-a7c2-2c20bae93cfc-ca-trust-extracted\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067306 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-installation-pull-secrets\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067306 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067268    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-certificates\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067306 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-image-registry-private-configuration\") pod \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\" (UID: \"c5128d14-493b-47eb-a7c2-2c20bae93cfc\") "
Jul 01 19:24:49.067587 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067499    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5128d14-493b-47eb-a7c2-2c20bae93cfc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:24:49.067788 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067701    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:49.067979 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.067933    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:49.069544 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.069522    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:49.069661 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.069562    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:49.069744 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.069722    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:49.069799 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.069764    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-kube-api-access-5swdv" (OuterVolumeSpecName: "kube-api-access-5swdv") pod "c5128d14-493b-47eb-a7c2-2c20bae93cfc" (UID: "c5128d14-493b-47eb-a7c2-2c20bae93cfc"). InnerVolumeSpecName "kube-api-access-5swdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:49.168200 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168164    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swdv\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-kube-api-access-5swdv\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.168200 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168189    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-bound-sa-token\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.168200 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168200    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5128d14-493b-47eb-a7c2-2c20bae93cfc-ca-trust-extracted\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.168200 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168208    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-installation-pull-secrets\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.168435 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168216    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-certificates\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.168435 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168227    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5128d14-493b-47eb-a7c2-2c20bae93cfc-image-registry-private-configuration\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.168435 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.168235    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5128d14-493b-47eb-a7c2-2c20bae93cfc-trusted-ca\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:49.984664 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.981174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845bc7bdd6-5cfw2"
Jul 01 19:24:49.984664 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.984614    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5xzb7"]
Jul 01 19:24:49.988942 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.988919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:49.991594 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.991575    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 19:24:49.992373 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.992294    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 19:24:49.992373 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.992310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 19:24:49.992373 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.992369    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tdcln\""
Jul 01 19:24:49.992574 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.992378    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 19:24:49.992574 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.992407    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 19:24:49.992710 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:49.992694    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 19:24:50.065257 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.065226    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-845bc7bdd6-5cfw2"]
Jul 01 19:24:50.074943 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.074908    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-845bc7bdd6-5cfw2"]
Jul 01 19:24:50.075081 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.074916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-wtmp\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075081 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.074981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-textfile\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075081 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-tls\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075081 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7gv\" (UniqueName: \"kubernetes.io/projected/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-kube-api-access-7z7gv\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075081 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075267 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-root\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075267 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075267 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-sys\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.075267 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.075193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-metrics-client-ca\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.175913 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.175879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-sys\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.175913 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.175913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-metrics-client-ca\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.175968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-wtmp\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.175990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-textfile\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-tls\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-sys\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7gv\" (UniqueName: \"kubernetes.io/projected/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-kube-api-access-7z7gv\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176122    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-wtmp\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176144 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-root\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176187    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-root\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176501 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:50.176202    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 19:24:50.176501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176267    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5128d14-493b-47eb-a7c2-2c20bae93cfc-registry-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:24:50.176501 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:50.176297    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-tls podName:785951ec-d3df-4e9b-89b4-06e66ef5dfe0 nodeName:}" failed. No retries permitted until 2026-07-01 19:24:50.676274632 +0000 UTC m=+155.777162870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-tls") pod "node-exporter-5xzb7" (UID: "785951ec-d3df-4e9b-89b4-06e66ef5dfe0") : secret "node-exporter-tls" not found
Jul 01 19:24:50.176501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176358    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-textfile\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176722 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-metrics-client-ca\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.176722 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.176705    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.178474 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.178453    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.186501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.186475    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7gv\" (UniqueName: \"kubernetes.io/projected/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-kube-api-access-7z7gv\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.680323 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.680291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-tls\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.682497 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.682466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/785951ec-d3df-4e9b-89b4-06e66ef5dfe0-node-exporter-tls\") pod \"node-exporter-5xzb7\" (UID: \"785951ec-d3df-4e9b-89b4-06e66ef5dfe0\") " pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.897536 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.897503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5xzb7"
Jul 01 19:24:50.909482 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:24:50.909455    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785951ec_d3df_4e9b_89b4_06e66ef5dfe0.slice/crio-32982736097d8f4c4f0fd9443bad6855e984484f5177fcc4525fe597ba2457f4 WatchSource:0}: Error finding container 32982736097d8f4c4f0fd9443bad6855e984484f5177fcc4525fe597ba2457f4: Status 404 returned error can't find the container with id 32982736097d8f4c4f0fd9443bad6855e984484f5177fcc4525fe597ba2457f4
Jul 01 19:24:50.985670 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:50.985620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5xzb7" event={"ID":"785951ec-d3df-4e9b-89b4-06e66ef5dfe0","Type":"ContainerStarted","Data":"32982736097d8f4c4f0fd9443bad6855e984484f5177fcc4525fe597ba2457f4"}
Jul 01 19:24:51.444508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:51.444471    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5128d14-493b-47eb-a7c2-2c20bae93cfc" path="/var/lib/kubelet/pods/c5128d14-493b-47eb-a7c2-2c20bae93cfc/volumes"
Jul 01 19:24:51.991101 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:51.991069    2578 generic.go:356] "Generic (PLEG): container finished" podID="785951ec-d3df-4e9b-89b4-06e66ef5dfe0" containerID="279570428e03c4e4524ec8b47d7a8d381554e1526900eba94d31dbe77e6c2f94" exitCode=0
Jul 01 19:24:51.991476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:51.991122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5xzb7" event={"ID":"785951ec-d3df-4e9b-89b4-06e66ef5dfe0","Type":"ContainerDied","Data":"279570428e03c4e4524ec8b47d7a8d381554e1526900eba94d31dbe77e6c2f94"}
Jul 01 19:24:52.997450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:52.997416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5xzb7" event={"ID":"785951ec-d3df-4e9b-89b4-06e66ef5dfe0","Type":"ContainerStarted","Data":"832c6c5c4a774f194703fa84768f67f3c60c850fa3ade906ea3cc38d2d17787c"}
Jul 01 19:24:52.997933 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:52.997463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5xzb7" event={"ID":"785951ec-d3df-4e9b-89b4-06e66ef5dfe0","Type":"ContainerStarted","Data":"bdf4457221856b914c6f9ee51251187d0127a21684b4eda8d8438ee91fbbce61"}
Jul 01 19:24:54.385625 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:24:54.385580    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dtzqg" podUID="94bd21a2-02a8-4572-bb24-c096e393aa8d"
Jul 01 19:24:55.003511 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:55.003483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:24:59.341942 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:59.341904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:24:59.344218 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:59.344194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94bd21a2-02a8-4572-bb24-c096e393aa8d-metrics-tls\") pod \"dns-default-dtzqg\" (UID: \"94bd21a2-02a8-4572-bb24-c096e393aa8d\") " pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:24:59.506917 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:59.506886    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2b6mr\""
Jul 01 19:24:59.514775 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:59.514750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:24:59.642459 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:59.642404    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5xzb7" podStartSLOduration=9.663364515 podStartE2EDuration="10.642386904s" totalImagesPullingTime="913.275766ms" totalInitContainerRuntime="65.746623ms" isStatefulPod=true podCreationTimestamp="2026-07-01 19:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:53.022374061 +0000 UTC m=+158.123262325" watchObservedRunningTime="2026-07-01 19:24:59.642386904 +0000 UTC m=+164.743275164"
Jul 01 19:24:59.643319 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:24:59.643291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dtzqg"]
Jul 01 19:24:59.646473 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:24:59.646450    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bd21a2_02a8_4572_bb24_c096e393aa8d.slice/crio-de6fe4abeebb11d32ee328141147a521290f68830eb9c0eb94bb6689d6528b2f WatchSource:0}: Error finding container de6fe4abeebb11d32ee328141147a521290f68830eb9c0eb94bb6689d6528b2f: Status 404 returned error can't find the container with id de6fe4abeebb11d32ee328141147a521290f68830eb9c0eb94bb6689d6528b2f
Jul 01 19:25:00.023452 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:00.023418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dtzqg" event={"ID":"94bd21a2-02a8-4572-bb24-c096e393aa8d","Type":"ContainerStarted","Data":"de6fe4abeebb11d32ee328141147a521290f68830eb9c0eb94bb6689d6528b2f"}
Jul 01 19:25:02.032363 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:02.032327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dtzqg" event={"ID":"94bd21a2-02a8-4572-bb24-c096e393aa8d","Type":"ContainerStarted","Data":"f0a4c2a0923fca8a35b1452e948ca6fc91d0598f94c01006a40e4c1bdc347d4e"}
Jul 01 19:25:02.032822 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:02.032374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dtzqg" event={"ID":"94bd21a2-02a8-4572-bb24-c096e393aa8d","Type":"ContainerStarted","Data":"61f7d8a85b9ebc8b54593c6440940f10e3c9724ed4c5fc432d2b08067b01b62e"}
Jul 01 19:25:02.032822 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:02.032431    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:25:02.055926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:02.055383    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dtzqg" podStartSLOduration=129.52642815 podStartE2EDuration="2m11.055362323s" totalImagesPullingTime="1.528934173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:25:02.054956873 +0000 UTC m=+167.155845133" watchObservedRunningTime="2026-07-01 19:25:02.055362323 +0000 UTC m=+167.156250583"
Jul 01 19:25:12.038644 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:12.038552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dtzqg"
Jul 01 19:25:31.650223 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:31.649874    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" podUID="ad89b138-a347-40b0-bb34-5c40e054b12a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 19:25:41.646953 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:41.646909    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" podUID="ad89b138-a347-40b0-bb34-5c40e054b12a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 19:25:51.646742 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:51.646690    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" podUID="ad89b138-a347-40b0-bb34-5c40e054b12a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 19:25:51.647092 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:51.646792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4"
Jul 01 19:25:51.647456 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:51.647413    2578 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8c66101ae21ef1d3031996c89d5b60e9f669f44227d333aea0e8f14b9ff1335b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 01 19:25:51.647518 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:51.647495    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" podUID="ad89b138-a347-40b0-bb34-5c40e054b12a" containerName="service-proxy" containerID="cri-o://8c66101ae21ef1d3031996c89d5b60e9f669f44227d333aea0e8f14b9ff1335b" gracePeriod=30
Jul 01 19:25:52.206835 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:52.206799    2578 generic.go:356] "Generic (PLEG): container finished" podID="ad89b138-a347-40b0-bb34-5c40e054b12a" containerID="8c66101ae21ef1d3031996c89d5b60e9f669f44227d333aea0e8f14b9ff1335b" exitCode=2
Jul 01 19:25:52.206835 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:52.206840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" event={"ID":"ad89b138-a347-40b0-bb34-5c40e054b12a","Type":"ContainerDied","Data":"8c66101ae21ef1d3031996c89d5b60e9f669f44227d333aea0e8f14b9ff1335b"}
Jul 01 19:25:52.207042 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:25:52.206870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b6d74ddb-lz2k4" event={"ID":"ad89b138-a347-40b0-bb34-5c40e054b12a","Type":"ContainerStarted","Data":"e8a107848e91638c9300228b8d2fa1bd03ae3fc2063c60364c12aef2b03c06dc"}
Jul 01 19:27:15.380057 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:27:15.380028    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 19:28:30.928356 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.928282    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-grbck"]
Jul 01 19:28:30.931290 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.931274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:30.936600 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.936571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-zrhzb\""
Jul 01 19:28:30.936600 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.936588    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 19:28:30.936801 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.936620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 01 19:28:30.936801 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.936666    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 19:28:30.951697 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:30.951671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-grbck"]
Jul 01 19:28:31.112926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.112898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1045a5fa-9844-43ca-b069-0bb3cc320045-cert\") pod \"kserve-controller-manager-79d94fb575-grbck\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") " pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.113095 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.112942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vldx\" (UniqueName: \"kubernetes.io/projected/1045a5fa-9844-43ca-b069-0bb3cc320045-kube-api-access-6vldx\") pod \"kserve-controller-manager-79d94fb575-grbck\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") " pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.214319 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.214225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1045a5fa-9844-43ca-b069-0bb3cc320045-cert\") pod \"kserve-controller-manager-79d94fb575-grbck\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") " pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.214319 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.214289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vldx\" (UniqueName: \"kubernetes.io/projected/1045a5fa-9844-43ca-b069-0bb3cc320045-kube-api-access-6vldx\") pod \"kserve-controller-manager-79d94fb575-grbck\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") " pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.216646 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.216607    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1045a5fa-9844-43ca-b069-0bb3cc320045-cert\") pod \"kserve-controller-manager-79d94fb575-grbck\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") " pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.228855 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.228822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vldx\" (UniqueName: \"kubernetes.io/projected/1045a5fa-9844-43ca-b069-0bb3cc320045-kube-api-access-6vldx\") pod \"kserve-controller-manager-79d94fb575-grbck\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") " pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.240719 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.240689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:31.371040 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.370945    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-grbck"]
Jul 01 19:28:31.373772 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:28:31.373744    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1045a5fa_9844_43ca_b069_0bb3cc320045.slice/crio-f68a3c73d56ba0855d0cfdcd0f3ffe5c31968ee87f0842aa3ca89034861d4363 WatchSource:0}: Error finding container f68a3c73d56ba0855d0cfdcd0f3ffe5c31968ee87f0842aa3ca89034861d4363: Status 404 returned error can't find the container with id f68a3c73d56ba0855d0cfdcd0f3ffe5c31968ee87f0842aa3ca89034861d4363
Jul 01 19:28:31.375133 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.375114    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:28:31.718266 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:31.718232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-grbck" event={"ID":"1045a5fa-9844-43ca-b069-0bb3cc320045","Type":"ContainerStarted","Data":"f68a3c73d56ba0855d0cfdcd0f3ffe5c31968ee87f0842aa3ca89034861d4363"}
Jul 01 19:28:35.732556 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:35.732522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-grbck" event={"ID":"1045a5fa-9844-43ca-b069-0bb3cc320045","Type":"ContainerStarted","Data":"8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8"}
Jul 01 19:28:35.732959 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:35.732615    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:28:35.752592 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:28:35.752546    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79d94fb575-grbck" podStartSLOduration=2.253002261 podStartE2EDuration="5.752530668s" totalImagesPullingTime="3.499528407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:35.750606619 +0000 UTC m=+380.851494877" watchObservedRunningTime="2026-07-01 19:28:35.752530668 +0000 UTC m=+380.853418926"
Jul 01 19:29:06.740720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:06.740690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:29:08.474657 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.474598    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-grbck"]
Jul 01 19:29:08.475128 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.474890    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-79d94fb575-grbck" podUID="1045a5fa-9844-43ca-b069-0bb3cc320045" containerName="manager" containerID="cri-o://8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8" gracePeriod=10
Jul 01 19:29:08.499898 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.499868    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-qmdwr"]
Jul 01 19:29:08.502801 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.502786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.513291 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.513264    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-qmdwr"]
Jul 01 19:29:08.577024 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.576986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwtx\" (UniqueName: \"kubernetes.io/projected/76e0cba6-2453-4e6e-9970-d54c216366ab-kube-api-access-8mwtx\") pod \"kserve-controller-manager-79d94fb575-qmdwr\" (UID: \"76e0cba6-2453-4e6e-9970-d54c216366ab\") " pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.577183 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.577037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76e0cba6-2453-4e6e-9970-d54c216366ab-cert\") pod \"kserve-controller-manager-79d94fb575-qmdwr\" (UID: \"76e0cba6-2453-4e6e-9970-d54c216366ab\") " pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.678081 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.678050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76e0cba6-2453-4e6e-9970-d54c216366ab-cert\") pod \"kserve-controller-manager-79d94fb575-qmdwr\" (UID: \"76e0cba6-2453-4e6e-9970-d54c216366ab\") " pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.678250 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.678114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwtx\" (UniqueName: \"kubernetes.io/projected/76e0cba6-2453-4e6e-9970-d54c216366ab-kube-api-access-8mwtx\") pod \"kserve-controller-manager-79d94fb575-qmdwr\" (UID: \"76e0cba6-2453-4e6e-9970-d54c216366ab\") " pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.680317 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.680291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76e0cba6-2453-4e6e-9970-d54c216366ab-cert\") pod \"kserve-controller-manager-79d94fb575-qmdwr\" (UID: \"76e0cba6-2453-4e6e-9970-d54c216366ab\") " pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.689227 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.689204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwtx\" (UniqueName: \"kubernetes.io/projected/76e0cba6-2453-4e6e-9970-d54c216366ab-kube-api-access-8mwtx\") pod \"kserve-controller-manager-79d94fb575-qmdwr\" (UID: \"76e0cba6-2453-4e6e-9970-d54c216366ab\") " pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.713576 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.713555    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:29:08.779297 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.779221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1045a5fa-9844-43ca-b069-0bb3cc320045-cert\") pod \"1045a5fa-9844-43ca-b069-0bb3cc320045\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") "
Jul 01 19:29:08.779297 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.779288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vldx\" (UniqueName: \"kubernetes.io/projected/1045a5fa-9844-43ca-b069-0bb3cc320045-kube-api-access-6vldx\") pod \"1045a5fa-9844-43ca-b069-0bb3cc320045\" (UID: \"1045a5fa-9844-43ca-b069-0bb3cc320045\") "
Jul 01 19:29:08.781385 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.781355    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1045a5fa-9844-43ca-b069-0bb3cc320045-kube-api-access-6vldx" (OuterVolumeSpecName: "kube-api-access-6vldx") pod "1045a5fa-9844-43ca-b069-0bb3cc320045" (UID: "1045a5fa-9844-43ca-b069-0bb3cc320045"). InnerVolumeSpecName "kube-api-access-6vldx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:29:08.781385 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.781376    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1045a5fa-9844-43ca-b069-0bb3cc320045-cert" (OuterVolumeSpecName: "cert") pod "1045a5fa-9844-43ca-b069-0bb3cc320045" (UID: "1045a5fa-9844-43ca-b069-0bb3cc320045"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:29:08.842299 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.842265    2578 generic.go:356] "Generic (PLEG): container finished" podID="1045a5fa-9844-43ca-b069-0bb3cc320045" containerID="8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8" exitCode=0
Jul 01 19:29:08.842463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.842309    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-grbck"
Jul 01 19:29:08.842463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.842354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-grbck" event={"ID":"1045a5fa-9844-43ca-b069-0bb3cc320045","Type":"ContainerDied","Data":"8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8"}
Jul 01 19:29:08.842463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.842408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-grbck" event={"ID":"1045a5fa-9844-43ca-b069-0bb3cc320045","Type":"ContainerDied","Data":"f68a3c73d56ba0855d0cfdcd0f3ffe5c31968ee87f0842aa3ca89034861d4363"}
Jul 01 19:29:08.842463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.842438    2578 scope.go:117] "RemoveContainer" containerID="8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8"
Jul 01 19:29:08.843958 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.843943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:08.853066 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.853051    2578 scope.go:117] "RemoveContainer" containerID="8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8"
Jul 01 19:29:08.853350 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:29:08.853332    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8\": container with ID starting with 8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8 not found: ID does not exist" containerID="8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8"
Jul 01 19:29:08.853399 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.853360    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8"} err="failed to get container status \"8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8\": rpc error: code = NotFound desc = could not find container \"8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8\": container with ID starting with 8fd0d9bf82b9cfe65f4a9d24c3df2dd0d484b65a0945fc68e579fc535b8a64f8 not found: ID does not exist"
Jul 01 19:29:08.865364 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.865340    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-grbck"]
Jul 01 19:29:08.868002 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.867981    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-grbck"]
Jul 01 19:29:08.880125 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.880101    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vldx\" (UniqueName: \"kubernetes.io/projected/1045a5fa-9844-43ca-b069-0bb3cc320045-kube-api-access-6vldx\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:29:08.880125 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.880123    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1045a5fa-9844-43ca-b069-0bb3cc320045-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:29:08.981076 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:08.981043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79d94fb575-qmdwr"]
Jul 01 19:29:08.985001 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:29:08.984974    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e0cba6_2453_4e6e_9970_d54c216366ab.slice/crio-0fdd45c9fbed73ab0a3f4fc1d8c24009980c474534fd387cb4f351be98378eaa WatchSource:0}: Error finding container 0fdd45c9fbed73ab0a3f4fc1d8c24009980c474534fd387cb4f351be98378eaa: Status 404 returned error can't find the container with id 0fdd45c9fbed73ab0a3f4fc1d8c24009980c474534fd387cb4f351be98378eaa
Jul 01 19:29:09.443560 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:09.443529    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1045a5fa-9844-43ca-b069-0bb3cc320045" path="/var/lib/kubelet/pods/1045a5fa-9844-43ca-b069-0bb3cc320045/volumes"
Jul 01 19:29:09.846895 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:09.846802    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr" event={"ID":"76e0cba6-2453-4e6e-9970-d54c216366ab","Type":"ContainerStarted","Data":"b070df16410031978e585d46ea35c2082677a086ea772585dc2cebe7d9c6bdb1"}
Jul 01 19:29:09.846895 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:09.846851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr" event={"ID":"76e0cba6-2453-4e6e-9970-d54c216366ab","Type":"ContainerStarted","Data":"0fdd45c9fbed73ab0a3f4fc1d8c24009980c474534fd387cb4f351be98378eaa"}
Jul 01 19:29:09.846895 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:09.846891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:09.864341 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:09.864289    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr" podStartSLOduration=1.357663784 podStartE2EDuration="1.864274013s" totalImagesPullingTime="506.610229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:09.863880391 +0000 UTC m=+414.964768663" watchObservedRunningTime="2026-07-01 19:29:09.864274013 +0000 UTC m=+414.965162269"
Jul 01 19:29:40.856824 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:40.856744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79d94fb575-qmdwr"
Jul 01 19:29:41.862939 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.862905    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-kn8dc"]
Jul 01 19:29:41.863307 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.863139    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1045a5fa-9844-43ca-b069-0bb3cc320045" containerName="manager"
Jul 01 19:29:41.863307 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.863149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1045a5fa-9844-43ca-b069-0bb3cc320045" containerName="manager"
Jul 01 19:29:41.863307 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.863230    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1045a5fa-9844-43ca-b069-0bb3cc320045" containerName="manager"
Jul 01 19:29:41.866016 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.865999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:41.868297 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.868273    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-vbwlw\""
Jul 01 19:29:41.868434 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.868421    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 19:29:41.880139 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:41.880119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-kn8dc"]
Jul 01 19:29:42.020497 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.020462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-cert\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.020689 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.020517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c55\" (UniqueName: \"kubernetes.io/projected/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-kube-api-access-m7c55\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.121206 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.121119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c55\" (UniqueName: \"kubernetes.io/projected/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-kube-api-access-m7c55\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.121206 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.121179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-cert\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.121391 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:29:42.121271    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 01 19:29:42.121391 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:29:42.121337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-cert podName:78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef nodeName:}" failed. No retries permitted until 2026-07-01 19:29:42.621321155 +0000 UTC m=+447.722209391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-cert") pod "odh-model-controller-84b54b7f97-kn8dc" (UID: "78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef") : secret "odh-model-controller-webhook-cert" not found
Jul 01 19:29:42.145741 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.145708    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c55\" (UniqueName: \"kubernetes.io/projected/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-kube-api-access-m7c55\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.624393 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.624356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-cert\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.626871 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.626838    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef-cert\") pod \"odh-model-controller-84b54b7f97-kn8dc\" (UID: \"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef\") " pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.775925 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.775889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:42.915312 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.915282    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-kn8dc"]
Jul 01 19:29:42.918819 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:29:42.918788    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e3ce6b_8572_4a0d_b1f4_eebbe425c7ef.slice/crio-ad4afde38bc6f9da026aea2dd5ad31fe7249b75799d0efd56dbb412f45f79804 WatchSource:0}: Error finding container ad4afde38bc6f9da026aea2dd5ad31fe7249b75799d0efd56dbb412f45f79804: Status 404 returned error can't find the container with id ad4afde38bc6f9da026aea2dd5ad31fe7249b75799d0efd56dbb412f45f79804
Jul 01 19:29:42.959183 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:42.959151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-kn8dc" event={"ID":"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef","Type":"ContainerStarted","Data":"ad4afde38bc6f9da026aea2dd5ad31fe7249b75799d0efd56dbb412f45f79804"}
Jul 01 19:29:45.971596 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:45.971561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-kn8dc" event={"ID":"78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef","Type":"ContainerStarted","Data":"cd3c62ffd4aabb8bd7d184f8990a6aec31bc8bb430c283bf80db3940dcaba97e"}
Jul 01 19:29:45.972045 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:45.971708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:45.991804 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:45.991755    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-kn8dc" podStartSLOduration=2.419099284 podStartE2EDuration="4.991740548s" totalImagesPullingTime="2.572641264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:45.991123434 +0000 UTC m=+451.092011704" watchObservedRunningTime="2026-07-01 19:29:45.991740548 +0000 UTC m=+451.092628806"
Jul 01 19:29:56.978244 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:56.978211    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-kn8dc"
Jul 01 19:29:57.812046 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.812009    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nsdbp"]
Jul 01 19:29:57.815190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.815173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nsdbp"
Jul 01 19:29:57.817527 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.817502    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 19:29:57.817791 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.817771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ppnjn\""
Jul 01 19:29:57.823313 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.823293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nsdbp"]
Jul 01 19:29:57.933819 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.933778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8x9\" (UniqueName: \"kubernetes.io/projected/402f27b8-6cf5-4834-a7d3-a214419d46cf-kube-api-access-sk8x9\") pod \"s3-init-nsdbp\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") " pod="kserve/s3-init-nsdbp"
Jul 01 19:29:57.933999 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:57.933886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402f27b8-6cf5-4834-a7d3-a214419d46cf-model-cache\") pod \"s3-init-nsdbp\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") " pod="kserve/s3-init-nsdbp"
Jul 01 19:29:58.035208 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:58.035170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402f27b8-6cf5-4834-a7d3-a214419d46cf-model-cache\") pod \"s3-init-nsdbp\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") " pod="kserve/s3-init-nsdbp"
Jul 01 19:29:58.035208 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:58.035214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8x9\" (UniqueName: \"kubernetes.io/projected/402f27b8-6cf5-4834-a7d3-a214419d46cf-kube-api-access-sk8x9\") pod \"s3-init-nsdbp\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") " pod="kserve/s3-init-nsdbp"
Jul 01 19:29:58.035602 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:58.035544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402f27b8-6cf5-4834-a7d3-a214419d46cf-model-cache\") pod \"s3-init-nsdbp\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") " pod="kserve/s3-init-nsdbp"
Jul 01 19:29:58.044430 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:58.044400    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8x9\" (UniqueName: \"kubernetes.io/projected/402f27b8-6cf5-4834-a7d3-a214419d46cf-kube-api-access-sk8x9\") pod \"s3-init-nsdbp\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") " pod="kserve/s3-init-nsdbp"
Jul 01 19:29:58.124500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:58.124403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nsdbp"
Jul 01 19:29:58.254316 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:58.254288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nsdbp"]
Jul 01 19:29:58.257234 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:29:58.257210    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod402f27b8_6cf5_4834_a7d3_a214419d46cf.slice/crio-d28bdc0065493c9806846a71f78aaa0265ad76e9435fe962b477563c25775f98 WatchSource:0}: Error finding container d28bdc0065493c9806846a71f78aaa0265ad76e9435fe962b477563c25775f98: Status 404 returned error can't find the container with id d28bdc0065493c9806846a71f78aaa0265ad76e9435fe962b477563c25775f98
Jul 01 19:29:59.020493 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:29:59.020458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nsdbp" event={"ID":"402f27b8-6cf5-4834-a7d3-a214419d46cf","Type":"ContainerStarted","Data":"d28bdc0065493c9806846a71f78aaa0265ad76e9435fe962b477563c25775f98"}
Jul 01 19:30:03.040775 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:30:03.040671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nsdbp" event={"ID":"402f27b8-6cf5-4834-a7d3-a214419d46cf","Type":"ContainerStarted","Data":"2148c04e21c88006fa9f5270aa1f069aeaeac09a0d64834c3077c3e15ddf102a"}
Jul 01 19:30:05.048314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:30:05.048274    2578 generic.go:356] "Generic (PLEG): container finished" podID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerID="2148c04e21c88006fa9f5270aa1f069aeaeac09a0d64834c3077c3e15ddf102a" exitCode=0
Jul 01 19:30:05.048720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:30:05.048344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nsdbp" event={"ID":"402f27b8-6cf5-4834-a7d3-a214419d46cf","Type":"ContainerDied","Data":"2148c04e21c88006fa9f5270aa1f069aeaeac09a0d64834c3077c3e15ddf102a"}
Jul 01 19:30:10.070348 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:30:10.070312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nsdbp" event={"ID":"402f27b8-6cf5-4834-a7d3-a214419d46cf","Type":"ContainerStarted","Data":"f57019860e9eaac657e27793cd0d565cefdbe7a75816f9ff7fdaf60b70cbe652"}
Jul 01 19:30:10.088758 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:30:10.088705    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-nsdbp" podStartSLOduration=2.582014871 podStartE2EDuration="13.088687685s" totalImagesPullingTime="8.984321439s" totalInitContainerRuntime="1.522351375s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:30:10.086800657 +0000 UTC m=+475.187688914" watchObservedRunningTime="2026-07-01 19:30:10.088687685 +0000 UTC m=+475.189575943"
Jul 01 19:31:15.290046 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:15.290001    2578 generic.go:356] "Generic (PLEG): container finished" podID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerID="f57019860e9eaac657e27793cd0d565cefdbe7a75816f9ff7fdaf60b70cbe652" exitCode=0
Jul 01 19:31:15.290448 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:15.290072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nsdbp" event={"ID":"402f27b8-6cf5-4834-a7d3-a214419d46cf","Type":"ContainerDied","Data":"f57019860e9eaac657e27793cd0d565cefdbe7a75816f9ff7fdaf60b70cbe652"}
Jul 01 19:31:16.412193 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.412170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nsdbp"
Jul 01 19:31:16.528640 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.528607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402f27b8-6cf5-4834-a7d3-a214419d46cf-model-cache\") pod \"402f27b8-6cf5-4834-a7d3-a214419d46cf\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") "
Jul 01 19:31:16.528823 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.528695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk8x9\" (UniqueName: \"kubernetes.io/projected/402f27b8-6cf5-4834-a7d3-a214419d46cf-kube-api-access-sk8x9\") pod \"402f27b8-6cf5-4834-a7d3-a214419d46cf\" (UID: \"402f27b8-6cf5-4834-a7d3-a214419d46cf\") "
Jul 01 19:31:16.530854 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.530822    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402f27b8-6cf5-4834-a7d3-a214419d46cf-kube-api-access-sk8x9" (OuterVolumeSpecName: "kube-api-access-sk8x9") pod "402f27b8-6cf5-4834-a7d3-a214419d46cf" (UID: "402f27b8-6cf5-4834-a7d3-a214419d46cf"). InnerVolumeSpecName "kube-api-access-sk8x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:31:16.547361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.547296    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402f27b8-6cf5-4834-a7d3-a214419d46cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "402f27b8-6cf5-4834-a7d3-a214419d46cf" (UID: "402f27b8-6cf5-4834-a7d3-a214419d46cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:31:16.629650 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.629598    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/402f27b8-6cf5-4834-a7d3-a214419d46cf-model-cache\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:31:16.629650 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:16.629627    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8x9\" (UniqueName: \"kubernetes.io/projected/402f27b8-6cf5-4834-a7d3-a214419d46cf-kube-api-access-sk8x9\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:31:17.298546 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:17.298517    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nsdbp"
Jul 01 19:31:17.298750 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:17.298513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nsdbp" event={"ID":"402f27b8-6cf5-4834-a7d3-a214419d46cf","Type":"ContainerDied","Data":"d28bdc0065493c9806846a71f78aaa0265ad76e9435fe962b477563c25775f98"}
Jul 01 19:31:17.298750 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:17.298623    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28bdc0065493c9806846a71f78aaa0265ad76e9435fe962b477563c25775f98"
Jul 01 19:31:49.962787 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.962752    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"]
Jul 01 19:31:49.963259 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.962987    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerName="s3-init"
Jul 01 19:31:49.963259 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.962998    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerName="s3-init"
Jul 01 19:31:49.963259 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.963006    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerName="download-hf-model"
Jul 01 19:31:49.963259 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.963012    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerName="download-hf-model"
Jul 01 19:31:49.963259 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.963047    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="402f27b8-6cf5-4834-a7d3-a214419d46cf" containerName="s3-init"
Jul 01 19:31:49.965838 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.965820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:49.969037 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.969012    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-ppnjn\""
Jul 01 19:31:49.969037 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.969012    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 01 19:31:49.969194 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.969017    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 01 19:31:49.973576 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:49.973555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"]
Jul 01 19:31:50.054922 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.054879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e2e06435-0411-4c04-bb3f-41c66994d30e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.055098 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.054959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2e06435-0411-4c04-bb3f-41c66994d30e-data\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.055098 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.054991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hhh\" (UniqueName: \"kubernetes.io/projected/e2e06435-0411-4c04-bb3f-41c66994d30e-kube-api-access-b5hhh\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.156201 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.156148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2e06435-0411-4c04-bb3f-41c66994d30e-data\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.156201 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.156206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hhh\" (UniqueName: \"kubernetes.io/projected/e2e06435-0411-4c04-bb3f-41c66994d30e-kube-api-access-b5hhh\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.156394 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.156239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e2e06435-0411-4c04-bb3f-41c66994d30e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.156621 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.156600    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2e06435-0411-4c04-bb3f-41c66994d30e-data\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.158675 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.158657    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e2e06435-0411-4c04-bb3f-41c66994d30e-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.164653 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.164613    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hhh\" (UniqueName: \"kubernetes.io/projected/e2e06435-0411-4c04-bb3f-41c66994d30e-kube-api-access-b5hhh\") pod \"seaweedfs-tls-custom-8b44d767d-l4lkw\" (UID: \"e2e06435-0411-4c04-bb3f-41c66994d30e\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.275810 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.275722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"
Jul 01 19:31:50.400681 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.400625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-l4lkw"]
Jul 01 19:31:50.403155 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:31:50.403128    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e06435_0411_4c04_bb3f_41c66994d30e.slice/crio-106baeb94ea801c8acf2e3aaeb208b6da98133885ab4adcaad1f93ec290ef8f6 WatchSource:0}: Error finding container 106baeb94ea801c8acf2e3aaeb208b6da98133885ab4adcaad1f93ec290ef8f6: Status 404 returned error can't find the container with id 106baeb94ea801c8acf2e3aaeb208b6da98133885ab4adcaad1f93ec290ef8f6
Jul 01 19:31:50.413197 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:50.413175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw" event={"ID":"e2e06435-0411-4c04-bb3f-41c66994d30e","Type":"ContainerStarted","Data":"106baeb94ea801c8acf2e3aaeb208b6da98133885ab4adcaad1f93ec290ef8f6"}
Jul 01 19:31:54.432641 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.432588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw" event={"ID":"e2e06435-0411-4c04-bb3f-41c66994d30e","Type":"ContainerStarted","Data":"808a88ef05b9c3a78997370b815e5abff0e32d5a8632a594d332c51b61d0ae32"}
Jul 01 19:31:54.451619 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.451570    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-l4lkw" podStartSLOduration=2.35753715 podStartE2EDuration="5.451553419s" totalImagesPullingTime="3.094016269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:54.450120047 +0000 UTC m=+579.551008316" watchObservedRunningTime="2026-07-01 19:31:54.451553419 +0000 UTC m=+579.552441677"
Jul 01 19:31:54.749174 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.749095    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-gwww9"]
Jul 01 19:31:54.756013 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.755983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:54.760576 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.760552    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-gwww9"]
Jul 01 19:31:54.791255 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.791221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b027f-7604-4e31-9065-0dffb95bc0a9-model-cache\") pod \"s3-tls-init-custom-gwww9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") " pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:54.791382 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.791264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zl4\" (UniqueName: \"kubernetes.io/projected/188b027f-7604-4e31-9065-0dffb95bc0a9-kube-api-access-k5zl4\") pod \"s3-tls-init-custom-gwww9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") " pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:54.891595 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.891558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zl4\" (UniqueName: \"kubernetes.io/projected/188b027f-7604-4e31-9065-0dffb95bc0a9-kube-api-access-k5zl4\") pod \"s3-tls-init-custom-gwww9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") " pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:54.891762 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.891673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b027f-7604-4e31-9065-0dffb95bc0a9-model-cache\") pod \"s3-tls-init-custom-gwww9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") " pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:54.891998 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.891982    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b027f-7604-4e31-9065-0dffb95bc0a9-model-cache\") pod \"s3-tls-init-custom-gwww9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") " pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:54.900668 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:54.900615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zl4\" (UniqueName: \"kubernetes.io/projected/188b027f-7604-4e31-9065-0dffb95bc0a9-kube-api-access-k5zl4\") pod \"s3-tls-init-custom-gwww9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") " pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:55.065495 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:55.065407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:31:55.190358 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:55.190322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-gwww9"]
Jul 01 19:31:55.193386 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:31:55.193356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod188b027f_7604_4e31_9065_0dffb95bc0a9.slice/crio-7bb6e5f8e5388840504e254c88fc199d90fe7b4d5686715def6f0682e52141a8 WatchSource:0}: Error finding container 7bb6e5f8e5388840504e254c88fc199d90fe7b4d5686715def6f0682e52141a8: Status 404 returned error can't find the container with id 7bb6e5f8e5388840504e254c88fc199d90fe7b4d5686715def6f0682e52141a8
Jul 01 19:31:55.438116 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:55.438086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-gwww9" event={"ID":"188b027f-7604-4e31-9065-0dffb95bc0a9","Type":"ContainerStarted","Data":"9f9333d0b1a4b2e20d189bb5ca0f4eb60b00475471d8e17452c521cc9d286c41"}
Jul 01 19:31:55.438504 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:55.438131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-gwww9" event={"ID":"188b027f-7604-4e31-9065-0dffb95bc0a9","Type":"ContainerStarted","Data":"7bb6e5f8e5388840504e254c88fc199d90fe7b4d5686715def6f0682e52141a8"}
Jul 01 19:31:57.447155 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:57.447119    2578 generic.go:356] "Generic (PLEG): container finished" podID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerID="9f9333d0b1a4b2e20d189bb5ca0f4eb60b00475471d8e17452c521cc9d286c41" exitCode=0
Jul 01 19:31:57.447523 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:57.447192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-gwww9" event={"ID":"188b027f-7604-4e31-9065-0dffb95bc0a9","Type":"ContainerDied","Data":"9f9333d0b1a4b2e20d189bb5ca0f4eb60b00475471d8e17452c521cc9d286c41"}
Jul 01 19:31:58.452992 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:58.452961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-gwww9" event={"ID":"188b027f-7604-4e31-9065-0dffb95bc0a9","Type":"ContainerStarted","Data":"7cd1291e063615a9a0ff538f9a10af55bd262ca47d541118d109e41770322f98"}
Jul 01 19:31:58.471113 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:31:58.471068    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-gwww9" podStartSLOduration=2.616242574 podStartE2EDuration="4.471053291s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.854810717s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:58.469877472 +0000 UTC m=+583.570765729" watchObservedRunningTime="2026-07-01 19:31:58.471053291 +0000 UTC m=+583.571941549"
Jul 01 19:32:04.474463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:04.474431    2578 generic.go:356] "Generic (PLEG): container finished" podID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerID="7cd1291e063615a9a0ff538f9a10af55bd262ca47d541118d109e41770322f98" exitCode=0
Jul 01 19:32:04.474463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:04.474468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-gwww9" event={"ID":"188b027f-7604-4e31-9065-0dffb95bc0a9","Type":"ContainerDied","Data":"7cd1291e063615a9a0ff538f9a10af55bd262ca47d541118d109e41770322f98"}
Jul 01 19:32:05.611245 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.611221    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:32:05.677900 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.677863    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b027f-7604-4e31-9065-0dffb95bc0a9-model-cache\") pod \"188b027f-7604-4e31-9065-0dffb95bc0a9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") "
Jul 01 19:32:05.678065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.677913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5zl4\" (UniqueName: \"kubernetes.io/projected/188b027f-7604-4e31-9065-0dffb95bc0a9-kube-api-access-k5zl4\") pod \"188b027f-7604-4e31-9065-0dffb95bc0a9\" (UID: \"188b027f-7604-4e31-9065-0dffb95bc0a9\") "
Jul 01 19:32:05.679992 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.679960    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188b027f-7604-4e31-9065-0dffb95bc0a9-kube-api-access-k5zl4" (OuterVolumeSpecName: "kube-api-access-k5zl4") pod "188b027f-7604-4e31-9065-0dffb95bc0a9" (UID: "188b027f-7604-4e31-9065-0dffb95bc0a9"). InnerVolumeSpecName "kube-api-access-k5zl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:32:05.698597 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.698567    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188b027f-7604-4e31-9065-0dffb95bc0a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "188b027f-7604-4e31-9065-0dffb95bc0a9" (UID: "188b027f-7604-4e31-9065-0dffb95bc0a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:32:05.778990 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.778902    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188b027f-7604-4e31-9065-0dffb95bc0a9-model-cache\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:32:05.778990 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:05.778931    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5zl4\" (UniqueName: \"kubernetes.io/projected/188b027f-7604-4e31-9065-0dffb95bc0a9-kube-api-access-k5zl4\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:32:06.483241 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:06.483215    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-gwww9"
Jul 01 19:32:06.483405 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:06.483204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-gwww9" event={"ID":"188b027f-7604-4e31-9065-0dffb95bc0a9","Type":"ContainerDied","Data":"7bb6e5f8e5388840504e254c88fc199d90fe7b4d5686715def6f0682e52141a8"}
Jul 01 19:32:06.483405 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:06.483326    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bb6e5f8e5388840504e254c88fc199d90fe7b4d5686715def6f0682e52141a8"
Jul 01 19:32:07.095116 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.095082    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"]
Jul 01 19:32:07.095508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.095354    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerName="download-hf-model"
Jul 01 19:32:07.095508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.095365    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerName="download-hf-model"
Jul 01 19:32:07.095508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.095375    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerName="s3-tls-init-custom"
Jul 01 19:32:07.095508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.095381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerName="s3-tls-init-custom"
Jul 01 19:32:07.095508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.095427    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="188b027f-7604-4e31-9065-0dffb95bc0a9" containerName="s3-tls-init-custom"
Jul 01 19:32:07.101500 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.101481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.103890 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.103869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 01 19:32:07.104066 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.104047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 01 19:32:07.106495 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.106476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"]
Jul 01 19:32:07.190191 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.190153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba0e7eee-9e18-4dac-be6b-9484babfe941-data\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.190376 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.190198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ba0e7eee-9e18-4dac-be6b-9484babfe941-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.190376 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.190256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp2rr\" (UniqueName: \"kubernetes.io/projected/ba0e7eee-9e18-4dac-be6b-9484babfe941-kube-api-access-vp2rr\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.291523 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.291487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba0e7eee-9e18-4dac-be6b-9484babfe941-data\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.291696 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.291534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ba0e7eee-9e18-4dac-be6b-9484babfe941-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.291696 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.291568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp2rr\" (UniqueName: \"kubernetes.io/projected/ba0e7eee-9e18-4dac-be6b-9484babfe941-kube-api-access-vp2rr\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.291986 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.291962    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ba0e7eee-9e18-4dac-be6b-9484babfe941-data\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.294123 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.294095    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/ba0e7eee-9e18-4dac-be6b-9484babfe941-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.299783 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.299754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp2rr\" (UniqueName: \"kubernetes.io/projected/ba0e7eee-9e18-4dac-be6b-9484babfe941-kube-api-access-vp2rr\") pod \"seaweedfs-tls-serving-67b7477c6d-blkg4\" (UID: \"ba0e7eee-9e18-4dac-be6b-9484babfe941\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.411702 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.411671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"
Jul 01 19:32:07.537845 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:07.537689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-blkg4"]
Jul 01 19:32:07.540380 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:32:07.540347    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0e7eee_9e18_4dac_be6b_9484babfe941.slice/crio-c93b6a346d0c93abf613a6f65150dd5113c9c32813e3395140d8a4438163b53c WatchSource:0}: Error finding container c93b6a346d0c93abf613a6f65150dd5113c9c32813e3395140d8a4438163b53c: Status 404 returned error can't find the container with id c93b6a346d0c93abf613a6f65150dd5113c9c32813e3395140d8a4438163b53c
Jul 01 19:32:08.491328 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:08.491286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4" event={"ID":"ba0e7eee-9e18-4dac-be6b-9484babfe941","Type":"ContainerStarted","Data":"a3173b1a579d2a523efa8fc7fa785b8dde47962725750c474c1c34c426b76fb5"}
Jul 01 19:32:08.491778 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:08.491345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4" event={"ID":"ba0e7eee-9e18-4dac-be6b-9484babfe941","Type":"ContainerStarted","Data":"c93b6a346d0c93abf613a6f65150dd5113c9c32813e3395140d8a4438163b53c"}
Jul 01 19:32:08.508403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:08.508357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-blkg4" podStartSLOduration=1.182942034 podStartE2EDuration="1.508342036s" totalImagesPullingTime="325.400002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:08.507664708 +0000 UTC m=+593.608552966" watchObservedRunningTime="2026-07-01 19:32:08.508342036 +0000 UTC m=+593.609230314"
Jul 01 19:32:09.053531 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.053493    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-d9vbq"]
Jul 01 19:32:09.057920 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.057902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.065604 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.065570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-d9vbq"]
Jul 01 19:32:09.116621 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.116587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45315d8c-819d-45c3-a88f-d97e5f4b3166-model-cache\") pod \"s3-tls-init-serving-d9vbq\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") " pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.116804 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.116626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/45315d8c-819d-45c3-a88f-d97e5f4b3166-kube-api-access-p9nvc\") pod \"s3-tls-init-serving-d9vbq\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") " pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.217618 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.217584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45315d8c-819d-45c3-a88f-d97e5f4b3166-model-cache\") pod \"s3-tls-init-serving-d9vbq\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") " pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.217618 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.217621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/45315d8c-819d-45c3-a88f-d97e5f4b3166-kube-api-access-p9nvc\") pod \"s3-tls-init-serving-d9vbq\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") " pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.217960 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.217941    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45315d8c-819d-45c3-a88f-d97e5f4b3166-model-cache\") pod \"s3-tls-init-serving-d9vbq\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") " pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.226278 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.226257    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/45315d8c-819d-45c3-a88f-d97e5f4b3166-kube-api-access-p9nvc\") pod \"s3-tls-init-serving-d9vbq\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") " pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.368048 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.367958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:09.496481 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:09.496462    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-d9vbq"]
Jul 01 19:32:09.498997 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:32:09.498972    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45315d8c_819d_45c3_a88f_d97e5f4b3166.slice/crio-67dfc8d9e5125db1b5bc252ef402b3b3a139ed61d3f58efdaaeae266d03120af WatchSource:0}: Error finding container 67dfc8d9e5125db1b5bc252ef402b3b3a139ed61d3f58efdaaeae266d03120af: Status 404 returned error can't find the container with id 67dfc8d9e5125db1b5bc252ef402b3b3a139ed61d3f58efdaaeae266d03120af
Jul 01 19:32:10.501536 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:10.501501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d9vbq" event={"ID":"45315d8c-819d-45c3-a88f-d97e5f4b3166","Type":"ContainerStarted","Data":"5ac9ce09d8181c8b13678092d148bfe71afc98235520f71b9dbb18f282bd25ae"}
Jul 01 19:32:10.501974 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:10.501549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d9vbq" event={"ID":"45315d8c-819d-45c3-a88f-d97e5f4b3166","Type":"ContainerStarted","Data":"67dfc8d9e5125db1b5bc252ef402b3b3a139ed61d3f58efdaaeae266d03120af"}
Jul 01 19:32:11.506962 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:11.506876    2578 generic.go:356] "Generic (PLEG): container finished" podID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerID="5ac9ce09d8181c8b13678092d148bfe71afc98235520f71b9dbb18f282bd25ae" exitCode=0
Jul 01 19:32:11.507400 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:11.506960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d9vbq" event={"ID":"45315d8c-819d-45c3-a88f-d97e5f4b3166","Type":"ContainerDied","Data":"5ac9ce09d8181c8b13678092d148bfe71afc98235520f71b9dbb18f282bd25ae"}
Jul 01 19:32:12.514289 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:12.514252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d9vbq" event={"ID":"45315d8c-819d-45c3-a88f-d97e5f4b3166","Type":"ContainerStarted","Data":"29d8102ca9f154845453d39c9dae9269eb4a79ed33cd9c97714631cbdb5fcccd"}
Jul 01 19:32:12.533526 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:12.533470    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-d9vbq" podStartSLOduration=1.896155046 podStartE2EDuration="3.533451429s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.637296383s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:12.532252534 +0000 UTC m=+597.633140792" watchObservedRunningTime="2026-07-01 19:32:12.533451429 +0000 UTC m=+597.634339686"
Jul 01 19:32:17.534843 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:17.534756    2578 generic.go:356] "Generic (PLEG): container finished" podID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerID="29d8102ca9f154845453d39c9dae9269eb4a79ed33cd9c97714631cbdb5fcccd" exitCode=0
Jul 01 19:32:17.534843 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:17.534829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d9vbq" event={"ID":"45315d8c-819d-45c3-a88f-d97e5f4b3166","Type":"ContainerDied","Data":"29d8102ca9f154845453d39c9dae9269eb4a79ed33cd9c97714631cbdb5fcccd"}
Jul 01 19:32:18.677643 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.677604    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:18.794914 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.794824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45315d8c-819d-45c3-a88f-d97e5f4b3166-model-cache\") pod \"45315d8c-819d-45c3-a88f-d97e5f4b3166\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") "
Jul 01 19:32:18.794914 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.794869    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/45315d8c-819d-45c3-a88f-d97e5f4b3166-kube-api-access-p9nvc\") pod \"45315d8c-819d-45c3-a88f-d97e5f4b3166\" (UID: \"45315d8c-819d-45c3-a88f-d97e5f4b3166\") "
Jul 01 19:32:18.797015 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.796987    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45315d8c-819d-45c3-a88f-d97e5f4b3166-kube-api-access-p9nvc" (OuterVolumeSpecName: "kube-api-access-p9nvc") pod "45315d8c-819d-45c3-a88f-d97e5f4b3166" (UID: "45315d8c-819d-45c3-a88f-d97e5f4b3166"). InnerVolumeSpecName "kube-api-access-p9nvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:32:18.816853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.816794    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45315d8c-819d-45c3-a88f-d97e5f4b3166-model-cache" (OuterVolumeSpecName: "model-cache") pod "45315d8c-819d-45c3-a88f-d97e5f4b3166" (UID: "45315d8c-819d-45c3-a88f-d97e5f4b3166"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:32:18.896262 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.896225    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45315d8c-819d-45c3-a88f-d97e5f4b3166-model-cache\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:32:18.896262 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:18.896254    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/45315d8c-819d-45c3-a88f-d97e5f4b3166-kube-api-access-p9nvc\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:32:19.543529 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:19.543496    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d9vbq"
Jul 01 19:32:19.543715 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:19.543499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d9vbq" event={"ID":"45315d8c-819d-45c3-a88f-d97e5f4b3166","Type":"ContainerDied","Data":"67dfc8d9e5125db1b5bc252ef402b3b3a139ed61d3f58efdaaeae266d03120af"}
Jul 01 19:32:19.543715 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:19.543610    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67dfc8d9e5125db1b5bc252ef402b3b3a139ed61d3f58efdaaeae266d03120af"
Jul 01 19:32:31.267138 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.267100    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"]
Jul 01 19:32:31.267706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.267356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerName="download-hf-model"
Jul 01 19:32:31.267706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.267367    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerName="download-hf-model"
Jul 01 19:32:31.267706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.267374    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerName="s3-tls-init-serving"
Jul 01 19:32:31.267706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.267379    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerName="s3-tls-init-serving"
Jul 01 19:32:31.267706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.267423    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="45315d8c-819d-45c3-a88f-d97e5f4b3166" containerName="s3-tls-init-serving"
Jul 01 19:32:31.271468 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.271448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.274670 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.274620    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:32:31.274789 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.274622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:32:31.274789 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.274620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:32:31.274789 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.274650    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:32:31.276499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.276484    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 01 19:32:31.276825 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.276810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 01 19:32:31.277014 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.276995    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zft66\""
Jul 01 19:32:31.289476 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.289452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"]
Jul 01 19:32:31.391103 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.391036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.391271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.391114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtsx\" (UniqueName: \"kubernetes.io/projected/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-api-access-6dtsx\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.391271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.391140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.391271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.391185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.391271 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.391204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.491710 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.491664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.491710 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.491713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtsx\" (UniqueName: \"kubernetes.io/projected/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-api-access-6dtsx\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.491971 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.491737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.491971 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.491778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.491971 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.491793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.491971 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:32:31.491891    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 01 19:32:31.491971 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:32:31.491969    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls podName:30592ced-3f5c-4b0c-a200-d8fdd11a393c nodeName:}" failed. No retries permitted until 2026-07-01 19:32:31.991950305 +0000 UTC m=+617.092838556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls") pod "isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" (UID: "30592ced-3f5c-4b0c-a200-d8fdd11a393c") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 01 19:32:31.492237 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.492216    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.492361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.492344    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.492361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.492355    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.504272 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.504242    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtsx\" (UniqueName: \"kubernetes.io/projected/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-api-access-6dtsx\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.996436 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.996396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:31.998818 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:31.998790    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:32.181584 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:32.181551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:32.315475 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:32.315453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"]
Jul 01 19:32:32.317944 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:32:32.317916    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30592ced_3f5c_4b0c_a200_d8fdd11a393c.slice/crio-64ad1ed8d808cbd422bd4e5afd308fc08997a31fae003f81f440896d592466e1 WatchSource:0}: Error finding container 64ad1ed8d808cbd422bd4e5afd308fc08997a31fae003f81f440896d592466e1: Status 404 returned error can't find the container with id 64ad1ed8d808cbd422bd4e5afd308fc08997a31fae003f81f440896d592466e1
Jul 01 19:32:32.593499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:32.593420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerStarted","Data":"78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4"}
Jul 01 19:32:32.593499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:32.593486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerStarted","Data":"64ad1ed8d808cbd422bd4e5afd308fc08997a31fae003f81f440896d592466e1"}
Jul 01 19:32:36.615677 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:36.615646    2578 generic.go:356] "Generic (PLEG): container finished" podID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerID="78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4" exitCode=0
Jul 01 19:32:36.616035 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:36.615688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerDied","Data":"78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4"}
Jul 01 19:32:51.697044 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:51.696957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerStarted","Data":"de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc"}
Jul 01 19:32:54.711262 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:54.711227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerStarted","Data":"717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c"}
Jul 01 19:32:57.725730 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:57.725684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerStarted","Data":"4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111"}
Jul 01 19:32:57.726152 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:57.725880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:57.726152 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:57.725925    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:57.727124 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:57.727097    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:32:57.748916 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:57.748870    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podStartSLOduration=2.725450217 podStartE2EDuration="26.748856117s" totalImagesPullingTime="20.367094016s" totalInitContainerRuntime="3.656311884s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:57.747484321 +0000 UTC m=+642.848372579" watchObservedRunningTime="2026-07-01 19:32:57.748856117 +0000 UTC m=+642.849744385"
Jul 01 19:32:58.729146 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:58.729114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:58.729651 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:58.729297    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:32:58.730341 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:58.730312    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:32:58.733196 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:58.733178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:32:59.732557 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:59.732515    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:32:59.732977 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:32:59.732835    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:00.736071 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:00.736030    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:33:00.736480 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:00.736426    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:10.736511 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:10.736457    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:33:10.737005 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:10.736951    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:20.736704 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:20.736626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:33:20.737190 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:20.737035    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:30.736114 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:30.736064    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:33:30.736623 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:30.736588    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:40.736950 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:40.736906    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:33:40.737400 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:40.737368    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:33:50.736825 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:50.736780    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:33:50.737324 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:33:50.737302    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:00.736720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:00.736678    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:34:00.737257 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:00.737232    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:10.737434 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:10.737340    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:10.737926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:10.737517    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:16.386375 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.386332    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"]
Jul 01 19:34:16.386850 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.386694    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" containerID="cri-o://4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111" gracePeriod=30
Jul 01 19:34:16.386850 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.386691    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" containerID="cri-o://de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc" gracePeriod=30
Jul 01 19:34:16.386850 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.386822    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" containerID="cri-o://717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c" gracePeriod=30
Jul 01 19:34:16.411173 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.411145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"]
Jul 01 19:34:16.414567 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.414546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.416830 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.416811    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 01 19:34:16.416938 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.416812    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 01 19:34:16.424845 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.424822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"]
Jul 01 19:34:16.512424 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.512397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.512570 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.512433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.512570 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.512478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.512570 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.512560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqg8\" (UniqueName: \"kubernetes.io/projected/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-api-access-lgqg8\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.512715 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.512596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c176608-fd44-45d4-b7c0-60e9523f4cda-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.613961 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.613923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614110 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.613990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqg8\" (UniqueName: \"kubernetes.io/projected/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-api-access-lgqg8\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614110 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.614038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c176608-fd44-45d4-b7c0-60e9523f4cda-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614110 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.614083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614310 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.614110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:34:16.614230    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 01 19:34:16.614310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:34:16.614307    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls podName:2c176608-fd44-45d4-b7c0-60e9523f4cda nodeName:}" failed. No retries permitted until 2026-07-01 19:34:17.114283668 +0000 UTC m=+722.215171912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" (UID: "2c176608-fd44-45d4-b7c0-60e9523f4cda") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 01 19:34:16.614493 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.614469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c176608-fd44-45d4-b7c0-60e9523f4cda-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614800 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.614779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.614847 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.614779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:16.624859 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:16.624834    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqg8\" (UniqueName: \"kubernetes.io/projected/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-api-access-lgqg8\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:17.018643 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:17.018597    2578 generic.go:356] "Generic (PLEG): container finished" podID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerID="717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c" exitCode=2
Jul 01 19:34:17.018825 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:17.018670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerDied","Data":"717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c"}
Jul 01 19:34:17.117251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:17.117210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:17.119742 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:17.119721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:17.325325 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:17.325234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:17.462168 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:17.462010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"]
Jul 01 19:34:17.464816 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:34:17.464782    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c176608_fd44_45d4_b7c0_60e9523f4cda.slice/crio-42e7ef7fd4d082a89119df6f6b918bb691e0bfaf0748892768ff501a1d7ae5b4 WatchSource:0}: Error finding container 42e7ef7fd4d082a89119df6f6b918bb691e0bfaf0748892768ff501a1d7ae5b4: Status 404 returned error can't find the container with id 42e7ef7fd4d082a89119df6f6b918bb691e0bfaf0748892768ff501a1d7ae5b4
Jul 01 19:34:18.024050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:18.024017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerStarted","Data":"2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758"}
Jul 01 19:34:18.024218 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:18.024086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerStarted","Data":"42e7ef7fd4d082a89119df6f6b918bb691e0bfaf0748892768ff501a1d7ae5b4"}
Jul 01 19:34:18.729474 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:18.729429    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:34:20.736547 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:20.736494    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:34:20.737097 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:20.736821    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:21.039554 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:21.039515    2578 generic.go:356] "Generic (PLEG): container finished" podID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerID="de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc" exitCode=0
Jul 01 19:34:21.039722 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:21.039581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerDied","Data":"de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc"}
Jul 01 19:34:22.045417 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:22.045384    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerID="2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758" exitCode=0
Jul 01 19:34:22.045930 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:22.045463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerDied","Data":"2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758"}
Jul 01 19:34:23.051598 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.051559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerStarted","Data":"53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b"}
Jul 01 19:34:23.052029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.051627    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerStarted","Data":"22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646"}
Jul 01 19:34:23.052029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.051666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerStarted","Data":"4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038"}
Jul 01 19:34:23.052029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.051957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:23.052029 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.052017    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:23.052197 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.052047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:23.053582 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.053537    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:34:23.054235 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.054214    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:23.073040 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.073000    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podStartSLOduration=3.554039182 podStartE2EDuration="7.072985997s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.518946815s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:34:23.071671503 +0000 UTC m=+728.172559761" watchObservedRunningTime="2026-07-01 19:34:23.072985997 +0000 UTC m=+728.173874254"
Jul 01 19:34:23.730300 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:23.730258    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:34:24.055767 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:24.055679    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:34:24.056100 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:24.056040    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:28.729821 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:28.729775    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:34:28.730242 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:28.729890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:29.060827 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:29.060745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:34:29.061266 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:29.061234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:34:29.061575 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:29.061554    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:30.736583 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:30.736543    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:34:30.737077 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:30.737044    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:33.729569 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:33.729525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:34:38.729765 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:38.729717    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:34:39.061917 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:39.061815    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:34:39.062282 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:39.062259    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:40.736684 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:40.736625    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 19:34:40.737098 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:40.736793    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:40.737098 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:40.737029    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:40.737214 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:40.737155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:43.729784 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:43.729743    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 19:34:46.546085 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.546059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:46.658386 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls\") pod \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") "
Jul 01 19:34:46.658577 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-cabundle-cert\") pod \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") "
Jul 01 19:34:46.658577 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658422    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-rbac-proxy-sar-config\") pod \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") "
Jul 01 19:34:46.658577 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658443    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kserve-provision-location\") pod \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") "
Jul 01 19:34:46.658577 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dtsx\" (UniqueName: \"kubernetes.io/projected/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-api-access-6dtsx\") pod \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\" (UID: \"30592ced-3f5c-4b0c-a200-d8fdd11a393c\") "
Jul 01 19:34:46.658870 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658835    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30592ced-3f5c-4b0c-a200-d8fdd11a393c" (UID: "30592ced-3f5c-4b0c-a200-d8fdd11a393c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:34:46.658870 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658853    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30592ced-3f5c-4b0c-a200-d8fdd11a393c" (UID: "30592ced-3f5c-4b0c-a200-d8fdd11a393c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:34:46.658988 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.658859    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30592ced-3f5c-4b0c-a200-d8fdd11a393c" (UID: "30592ced-3f5c-4b0c-a200-d8fdd11a393c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:34:46.660706 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.660678    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-api-access-6dtsx" (OuterVolumeSpecName: "kube-api-access-6dtsx") pod "30592ced-3f5c-4b0c-a200-d8fdd11a393c" (UID: "30592ced-3f5c-4b0c-a200-d8fdd11a393c"). InnerVolumeSpecName "kube-api-access-6dtsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:34:46.660811 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.660759    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30592ced-3f5c-4b0c-a200-d8fdd11a393c" (UID: "30592ced-3f5c-4b0c-a200-d8fdd11a393c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:34:46.759763 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.759728    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:34:46.759763 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.759758    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:34:46.759763 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.759767    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:34:46.759994 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.759775    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtsx\" (UniqueName: \"kubernetes.io/projected/30592ced-3f5c-4b0c-a200-d8fdd11a393c-kube-api-access-6dtsx\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:34:46.759994 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:46.759783    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30592ced-3f5c-4b0c-a200-d8fdd11a393c-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:34:47.143260 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.143180    2578 generic.go:356] "Generic (PLEG): container finished" podID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerID="4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111" exitCode=0
Jul 01 19:34:47.143260 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.143223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerDied","Data":"4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111"}
Jul 01 19:34:47.143260 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.143238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"
Jul 01 19:34:47.143463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.143275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j" event={"ID":"30592ced-3f5c-4b0c-a200-d8fdd11a393c","Type":"ContainerDied","Data":"64ad1ed8d808cbd422bd4e5afd308fc08997a31fae003f81f440896d592466e1"}
Jul 01 19:34:47.143463 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.143306    2578 scope.go:117] "RemoveContainer" containerID="4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111"
Jul 01 19:34:47.151543 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.151527    2578 scope.go:117] "RemoveContainer" containerID="717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c"
Jul 01 19:34:47.160646 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.160591    2578 scope.go:117] "RemoveContainer" containerID="de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc"
Jul 01 19:34:47.168828 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.168809    2578 scope.go:117] "RemoveContainer" containerID="78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4"
Jul 01 19:34:47.175906 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.175890    2578 scope.go:117] "RemoveContainer" containerID="4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111"
Jul 01 19:34:47.176165 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:34:47.176146    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111\": container with ID starting with 4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111 not found: ID does not exist" containerID="4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111"
Jul 01 19:34:47.176221 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176174    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111"} err="failed to get container status \"4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111\": rpc error: code = NotFound desc = could not find container \"4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111\": container with ID starting with 4f7a480eb3045c83093f7be39a651e8fec1b2408c2eec8df617ed7e0c3bdb111 not found: ID does not exist"
Jul 01 19:34:47.176221 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176193    2578 scope.go:117] "RemoveContainer" containerID="717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c"
Jul 01 19:34:47.176415 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:34:47.176393    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c\": container with ID starting with 717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c not found: ID does not exist" containerID="717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c"
Jul 01 19:34:47.176475 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176422    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c"} err="failed to get container status \"717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c\": rpc error: code = NotFound desc = could not find container \"717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c\": container with ID starting with 717fa8157c2a91f24d96dc457b60d882006fab3e191d546fde15b7a027b2fb2c not found: ID does not exist"
Jul 01 19:34:47.176475 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176438    2578 scope.go:117] "RemoveContainer" containerID="de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc"
Jul 01 19:34:47.176690 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:34:47.176673    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc\": container with ID starting with de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc not found: ID does not exist" containerID="de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc"
Jul 01 19:34:47.176745 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc"} err="failed to get container status \"de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc\": rpc error: code = NotFound desc = could not find container \"de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc\": container with ID starting with de4eb3c80fcf08c3c89e5ba217312eb343e55eb8ad9d372e9034293910250cbc not found: ID does not exist"
Jul 01 19:34:47.176745 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176706    2578 scope.go:117] "RemoveContainer" containerID="78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4"
Jul 01 19:34:47.176963 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:34:47.176945    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4\": container with ID starting with 78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4 not found: ID does not exist" containerID="78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4"
Jul 01 19:34:47.176997 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.176975    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4"} err="failed to get container status \"78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4\": rpc error: code = NotFound desc = could not find container \"78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4\": container with ID starting with 78c00a6d06a2b83aa49f28fbc1cdfd2f57ccee03e6ce08479e9d944ebb376ea4 not found: ID does not exist"
Jul 01 19:34:47.203981 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.203954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"]
Jul 01 19:34:47.217989 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.217967    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-tv68j"]
Jul 01 19:34:47.448417 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:47.448386    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" path="/var/lib/kubelet/pods/30592ced-3f5c-4b0c-a200-d8fdd11a393c/volumes"
Jul 01 19:34:49.061651 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:49.061598    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:34:49.062157 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:49.062134    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:34:59.061709 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:59.061660    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:34:59.062147 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:34:59.062101    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:35:09.061492 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:09.061446    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:35:09.061941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:09.061910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:35:19.061408 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:19.061371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:35:19.061853 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:19.061784    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:35:29.061217 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:29.061173    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:35:29.061697 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:29.061578    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:35:30.440313 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:30.440268    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:35:30.440783 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:30.440695    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:35:40.440784 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:40.440708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:35:40.441138 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:40.440813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:35:51.594760 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.594729    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"]
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595125    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595143    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595159    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595167    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595179    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="storage-initializer"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595188    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="storage-initializer"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595200    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595277    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kube-rbac-proxy"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595289    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="agent"
Jul 01 19:35:51.595331 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.595301    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30592ced-3f5c-4b0c-a200-d8fdd11a393c" containerName="kserve-container"
Jul 01 19:35:51.598652 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.598607    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"]
Jul 01 19:35:51.598787 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.598767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.598979 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.598932    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" containerID="cri-o://4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038" gracePeriod=30
Jul 01 19:35:51.599103 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.599074    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" containerID="cri-o://22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646" gracePeriod=30
Jul 01 19:35:51.599182 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.599052    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" containerID="cri-o://53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b" gracePeriod=30
Jul 01 19:35:51.601327 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.601303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 01 19:35:51.601466 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.601442    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 01 19:35:51.608251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.608230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"]
Jul 01 19:35:51.749378 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.749347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.749557 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.749445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823baaa0-2b4e-49cd-8900-11ce6b54d01d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.749557 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.749484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x66w\" (UniqueName: \"kubernetes.io/projected/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-api-access-6x66w\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.850717 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.850596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.850717 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.850700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823baaa0-2b4e-49cd-8900-11ce6b54d01d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.850944 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.850739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x66w\" (UniqueName: \"kubernetes.io/projected/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-api-access-6x66w\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.851295 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.851270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.853053 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.853031    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823baaa0-2b4e-49cd-8900-11ce6b54d01d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.859404 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.859375    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x66w\" (UniqueName: \"kubernetes.io/projected/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-api-access-6x66w\") pod \"message-dumper-predictor-6575bcfdfd-gxcmx\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:51.917534 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:51.917497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:52.044711 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:52.044680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"]
Jul 01 19:35:52.047548 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:35:52.047517    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823baaa0_2b4e_49cd_8900_11ce6b54d01d.slice/crio-25e36be23eb819ed07c65480d17128fa5477fa3683710f742eaf9332b2528d6e WatchSource:0}: Error finding container 25e36be23eb819ed07c65480d17128fa5477fa3683710f742eaf9332b2528d6e: Status 404 returned error can't find the container with id 25e36be23eb819ed07c65480d17128fa5477fa3683710f742eaf9332b2528d6e
Jul 01 19:35:52.049373 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:52.049355    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:35:52.390049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:52.390008    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerID="22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646" exitCode=2
Jul 01 19:35:52.390239 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:52.390080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerDied","Data":"22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646"}
Jul 01 19:35:52.391173 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:52.391155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" event={"ID":"823baaa0-2b4e-49cd-8900-11ce6b54d01d","Type":"ContainerStarted","Data":"25e36be23eb819ed07c65480d17128fa5477fa3683710f742eaf9332b2528d6e"}
Jul 01 19:35:54.056350 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.056309    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:35:54.401072 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.400980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" event={"ID":"823baaa0-2b4e-49cd-8900-11ce6b54d01d","Type":"ContainerStarted","Data":"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"}
Jul 01 19:35:54.401072 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.401041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" event={"ID":"823baaa0-2b4e-49cd-8900-11ce6b54d01d","Type":"ContainerStarted","Data":"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"}
Jul 01 19:35:54.401253 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.401079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:54.401253 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.401121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:54.402823 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.402803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:35:54.419310 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:54.419264    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" podStartSLOduration=1.937448142 podStartE2EDuration="3.41924726s" totalImagesPullingTime="1.481799118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:35:54.418350616 +0000 UTC m=+819.519238909" watchObservedRunningTime="2026-07-01 19:35:54.41924726 +0000 UTC m=+819.520135519"
Jul 01 19:35:57.414313 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:57.414280    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerID="4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038" exitCode=0
Jul 01 19:35:57.414709 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:57.414344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerDied","Data":"4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038"}
Jul 01 19:35:59.056773 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:35:59.056730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:36:00.440158 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:00.440114    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:36:00.440552 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:00.440371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:01.412596 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:01.412569    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:36:04.055905 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:04.055863    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:36:04.056312 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:04.056007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:36:09.056801 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:09.056760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:36:10.440880 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:10.440830    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:36:10.441272 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:10.441197    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:11.632473 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.632435    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"]
Jul 01 19:36:11.636104 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.636083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.638574 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.638545    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 01 19:36:11.638734 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.638623    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 01 19:36:11.651517 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.651491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"]
Jul 01 19:36:11.814885 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.814850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.814885 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.814884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vr5d\" (UniqueName: \"kubernetes.io/projected/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-api-access-9vr5d\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.815089 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.814908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.815089 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.815034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.815089 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.815066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.916449 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.916420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.916673 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.916456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.916673 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.916493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.916812 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:36:11.916741    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 01 19:36:11.916812 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.916798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vr5d\" (UniqueName: \"kubernetes.io/projected/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-api-access-9vr5d\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.916906 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:36:11.916832    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls podName:94a7be8e-e234-4a30-9613-1f4f7c7cc378 nodeName:}" failed. No retries permitted until 2026-07-01 19:36:12.416809908 +0000 UTC m=+837.517698146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls") pod "isvc-logger-predictor-599b89d7c-4csn9" (UID: "94a7be8e-e234-4a30-9613-1f4f7c7cc378") : secret "isvc-logger-predictor-serving-cert" not found
Jul 01 19:36:11.916906 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.916886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.917299 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.917270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.917400 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.917275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.917400 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.917355    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:11.925457 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:11.925437    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vr5d\" (UniqueName: \"kubernetes.io/projected/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-api-access-9vr5d\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:12.421093 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:12.421061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:12.423420 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:12.423397    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-4csn9\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:12.546261 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:12.546222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:12.679582 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:12.679557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"]
Jul 01 19:36:12.681839 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:36:12.681814    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a7be8e_e234_4a30_9613_1f4f7c7cc378.slice/crio-686d60ec40f5ca89c918a0c73c6d503cc673b4b2e83a822de8e2107977b79ca3 WatchSource:0}: Error finding container 686d60ec40f5ca89c918a0c73c6d503cc673b4b2e83a822de8e2107977b79ca3: Status 404 returned error can't find the container with id 686d60ec40f5ca89c918a0c73c6d503cc673b4b2e83a822de8e2107977b79ca3
Jul 01 19:36:13.482092 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:13.482059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerStarted","Data":"ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee"}
Jul 01 19:36:13.482265 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:13.482126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerStarted","Data":"686d60ec40f5ca89c918a0c73c6d503cc673b4b2e83a822de8e2107977b79ca3"}
Jul 01 19:36:14.056026 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:14.055990    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:36:16.495065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:16.495026    2578 generic.go:356] "Generic (PLEG): container finished" podID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerID="ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee" exitCode=0
Jul 01 19:36:16.495425 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:16.495105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerDied","Data":"ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee"}
Jul 01 19:36:17.501687 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.501654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerStarted","Data":"ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1"}
Jul 01 19:36:17.502129 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.501719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerStarted","Data":"a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f"}
Jul 01 19:36:17.502129 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.501741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerStarted","Data":"9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564"}
Jul 01 19:36:17.502129 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.502041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:17.502129 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.502097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:17.502129 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.502134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:17.503590 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.503564    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:36:17.509001 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.508956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:17.526349 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:17.526297    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podStartSLOduration=3.000176156 podStartE2EDuration="6.526282908s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.526106752s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:36:17.524607532 +0000 UTC m=+842.625495790" watchObservedRunningTime="2026-07-01 19:36:17.526282908 +0000 UTC m=+842.627171166"
Jul 01 19:36:18.505484 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:18.505442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:36:18.505884 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:18.505844    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:19.056285 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:19.056245    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 01 19:36:20.440708 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:20.440666    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:5000: connect: connection refused"
Jul 01 19:36:20.441119 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:20.440819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:36:20.441119 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:20.440976    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:20.441119 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:20.441084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:36:21.742602 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.742575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:36:21.794327 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgqg8\" (UniqueName: \"kubernetes.io/projected/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-api-access-lgqg8\") pod \"2c176608-fd44-45d4-b7c0-60e9523f4cda\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") "
Jul 01 19:36:21.794522 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls\") pod \"2c176608-fd44-45d4-b7c0-60e9523f4cda\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") "
Jul 01 19:36:21.794522 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794360    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c176608-fd44-45d4-b7c0-60e9523f4cda-kserve-provision-location\") pod \"2c176608-fd44-45d4-b7c0-60e9523f4cda\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") "
Jul 01 19:36:21.794522 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-cabundle-cert\") pod \"2c176608-fd44-45d4-b7c0-60e9523f4cda\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") "
Jul 01 19:36:21.794522 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-rbac-proxy-sar-config\") pod \"2c176608-fd44-45d4-b7c0-60e9523f4cda\" (UID: \"2c176608-fd44-45d4-b7c0-60e9523f4cda\") "
Jul 01 19:36:21.794774 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794745    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c176608-fd44-45d4-b7c0-60e9523f4cda" (UID: "2c176608-fd44-45d4-b7c0-60e9523f4cda"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:21.794821 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794759    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c176608-fd44-45d4-b7c0-60e9523f4cda-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c176608-fd44-45d4-b7c0-60e9523f4cda" (UID: "2c176608-fd44-45d4-b7c0-60e9523f4cda"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:36:21.794915 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.794895    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c176608-fd44-45d4-b7c0-60e9523f4cda" (UID: "2c176608-fd44-45d4-b7c0-60e9523f4cda"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:36:21.796442 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.796416    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-api-access-lgqg8" (OuterVolumeSpecName: "kube-api-access-lgqg8") pod "2c176608-fd44-45d4-b7c0-60e9523f4cda" (UID: "2c176608-fd44-45d4-b7c0-60e9523f4cda"). InnerVolumeSpecName "kube-api-access-lgqg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:36:21.796530 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.796480    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c176608-fd44-45d4-b7c0-60e9523f4cda" (UID: "2c176608-fd44-45d4-b7c0-60e9523f4cda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:36:21.895086 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.895000    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqg8\" (UniqueName: \"kubernetes.io/projected/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-api-access-lgqg8\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:36:21.895086 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.895031    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c176608-fd44-45d4-b7c0-60e9523f4cda-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:36:21.895086 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.895041    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c176608-fd44-45d4-b7c0-60e9523f4cda-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:36:21.895086 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.895050    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:36:21.895086 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:21.895059    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c176608-fd44-45d4-b7c0-60e9523f4cda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:36:22.523361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.523324    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerID="53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b" exitCode=0
Jul 01 19:36:22.523556 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.523395    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"
Jul 01 19:36:22.523556 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.523412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerDied","Data":"53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b"}
Jul 01 19:36:22.523556 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.523495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr" event={"ID":"2c176608-fd44-45d4-b7c0-60e9523f4cda","Type":"ContainerDied","Data":"42e7ef7fd4d082a89119df6f6b918bb691e0bfaf0748892768ff501a1d7ae5b4"}
Jul 01 19:36:22.523556 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.523539    2578 scope.go:117] "RemoveContainer" containerID="53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b"
Jul 01 19:36:22.531964 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.531944    2578 scope.go:117] "RemoveContainer" containerID="22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646"
Jul 01 19:36:22.539126 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.539109    2578 scope.go:117] "RemoveContainer" containerID="4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038"
Jul 01 19:36:22.546009 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.545983    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"]
Jul 01 19:36:22.547237 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.547218    2578 scope.go:117] "RemoveContainer" containerID="2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758"
Jul 01 19:36:22.554649 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.550614    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-4w9cr"]
Jul 01 19:36:22.557079 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557064    2578 scope.go:117] "RemoveContainer" containerID="53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b"
Jul 01 19:36:22.557336 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:36:22.557318    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b\": container with ID starting with 53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b not found: ID does not exist" containerID="53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b"
Jul 01 19:36:22.557388 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557344    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b"} err="failed to get container status \"53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b\": rpc error: code = NotFound desc = could not find container \"53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b\": container with ID starting with 53220b4dadb2583cea1af1f4b057c32fcd408740693787eac72512fc71adc99b not found: ID does not exist"
Jul 01 19:36:22.557388 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557362    2578 scope.go:117] "RemoveContainer" containerID="22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646"
Jul 01 19:36:22.557618 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:36:22.557601    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646\": container with ID starting with 22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646 not found: ID does not exist" containerID="22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646"
Jul 01 19:36:22.557677 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557623    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646"} err="failed to get container status \"22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646\": rpc error: code = NotFound desc = could not find container \"22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646\": container with ID starting with 22a21333a3d83de53631d6b453aa98ffc017e4815d2edd04a32e8c85920ec646 not found: ID does not exist"
Jul 01 19:36:22.557677 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557655    2578 scope.go:117] "RemoveContainer" containerID="4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038"
Jul 01 19:36:22.557860 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:36:22.557847    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038\": container with ID starting with 4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038 not found: ID does not exist" containerID="4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038"
Jul 01 19:36:22.557941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557862    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038"} err="failed to get container status \"4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038\": rpc error: code = NotFound desc = could not find container \"4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038\": container with ID starting with 4ccc4a870374aafeeb2a160873f6441d776d73c9ce811c1ffe8238a19e314038 not found: ID does not exist"
Jul 01 19:36:22.557941 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.557874    2578 scope.go:117] "RemoveContainer" containerID="2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758"
Jul 01 19:36:22.558083 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:36:22.558068    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758\": container with ID starting with 2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758 not found: ID does not exist" containerID="2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758"
Jul 01 19:36:22.558135 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:22.558087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758"} err="failed to get container status \"2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758\": rpc error: code = NotFound desc = could not find container \"2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758\": container with ID starting with 2c85e850aca2ce2455e13e2658d72e7f24f63e25139ca3fb989e0d95e039c758 not found: ID does not exist"
Jul 01 19:36:23.444233 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:23.444203    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" path="/var/lib/kubelet/pods/2c176608-fd44-45d4-b7c0-60e9523f4cda/volumes"
Jul 01 19:36:23.510746 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:23.510719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:36:23.511467 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:23.511432    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:36:23.511693 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:23.511669    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:33.511357 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:33.511312    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:36:33.511860 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:33.511837    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:43.511336 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:43.511292    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:36:43.511834 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:43.511806    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:36:53.511478 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:53.511431    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:36:53.512071 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:36:53.512013    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:03.511864 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:03.511821    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:37:03.512370 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:03.512271    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:13.511960 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:13.511866    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:37:13.512399 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:13.512359    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:23.511693 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:23.511626    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:37:23.512156 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:23.512133    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:24.439599 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:24.439546    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:37:24.439911 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:24.439881    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:34.439840 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:34.439809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:37:34.440385 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:34.440364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:37:46.651677 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.651648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-gxcmx_823baaa0-2b4e-49cd-8900-11ce6b54d01d/kserve-container/0.log"
Jul 01 19:37:46.932060 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932025    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"]
Jul 01 19:37:46.932314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932300    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container"
Jul 01 19:37:46.932314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932313    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932334    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932341    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="storage-initializer"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932354    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="storage-initializer"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932361    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932366    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932411    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kube-rbac-proxy"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932419    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="agent"
Jul 01 19:37:46.932450 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.932425    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c176608-fd44-45d4-b7c0-60e9523f4cda" containerName="kserve-container"
Jul 01 19:37:46.935467 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.935445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:46.937914 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.937894    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 01 19:37:46.938165 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.938146    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 01 19:37:46.946937 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.946912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"]
Jul 01 19:37:46.967872 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.967841    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"]
Jul 01 19:37:46.968167 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.968101    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" containerID="cri-o://9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564" gracePeriod=30
Jul 01 19:37:46.968231 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.968182    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" containerID="cri-o://a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f" gracePeriod=30
Jul 01 19:37:46.968299 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:46.968178    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" containerID="cri-o://ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1" gracePeriod=30
Jul 01 19:37:47.010374 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.010327    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"]
Jul 01 19:37:47.010618 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.010575    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kserve-container" containerID="cri-o://fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6" gracePeriod=30
Jul 01 19:37:47.010730 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.010604    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kube-rbac-proxy" containerID="cri-o://1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88" gracePeriod=30
Jul 01 19:37:47.048457 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.048424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.048457 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.048461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.048662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.048482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.048662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.048547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4q7\" (UniqueName: \"kubernetes.io/projected/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-api-access-nx4q7\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.048662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.048590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.149925 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.149881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150092 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.149980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150092 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.150010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150227 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.150089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150227 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.150172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4q7\" (UniqueName: \"kubernetes.io/projected/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-api-access-nx4q7\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150532 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.150509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150721 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.150701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.150782 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.150754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.152568 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.152550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.160109 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.160087    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4q7\" (UniqueName: \"kubernetes.io/projected/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-api-access-nx4q7\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.248007 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.247984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:37:47.251255 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.251236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:37:47.352305 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.352276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-rbac-proxy-sar-config\") pod \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") "
Jul 01 19:37:47.352465 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.352324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x66w\" (UniqueName: \"kubernetes.io/projected/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-api-access-6x66w\") pod \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") "
Jul 01 19:37:47.352465 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.352355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823baaa0-2b4e-49cd-8900-11ce6b54d01d-proxy-tls\") pod \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\" (UID: \"823baaa0-2b4e-49cd-8900-11ce6b54d01d\") "
Jul 01 19:37:47.352738 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.352709    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "823baaa0-2b4e-49cd-8900-11ce6b54d01d" (UID: "823baaa0-2b4e-49cd-8900-11ce6b54d01d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:37:47.354963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.354936    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823baaa0-2b4e-49cd-8900-11ce6b54d01d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "823baaa0-2b4e-49cd-8900-11ce6b54d01d" (UID: "823baaa0-2b4e-49cd-8900-11ce6b54d01d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:37:47.355116 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.355092    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-api-access-6x66w" (OuterVolumeSpecName: "kube-api-access-6x66w") pod "823baaa0-2b4e-49cd-8900-11ce6b54d01d" (UID: "823baaa0-2b4e-49cd-8900-11ce6b54d01d"). InnerVolumeSpecName "kube-api-access-6x66w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:37:47.389019 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.388839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"]
Jul 01 19:37:47.391096 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:37:47.391072    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c90f011_879c_48e4_b3cd_ff2c43b8c8a9.slice/crio-234901cfab937bad1a145560841d1512d9e6d6f6dcbf98b1747910b19617ea66 WatchSource:0}: Error finding container 234901cfab937bad1a145560841d1512d9e6d6f6dcbf98b1747910b19617ea66: Status 404 returned error can't find the container with id 234901cfab937bad1a145560841d1512d9e6d6f6dcbf98b1747910b19617ea66
Jul 01 19:37:47.453151 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.453129    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/823baaa0-2b4e-49cd-8900-11ce6b54d01d-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:37:47.453151 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.453153    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:37:47.453329 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.453162    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x66w\" (UniqueName: \"kubernetes.io/projected/823baaa0-2b4e-49cd-8900-11ce6b54d01d-kube-api-access-6x66w\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:37:47.845946 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.845908    2578 generic.go:356] "Generic (PLEG): container finished" podID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerID="a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f" exitCode=2
Jul 01 19:37:47.846435 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.845980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerDied","Data":"a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f"}
Jul 01 19:37:47.847479 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.847442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerStarted","Data":"a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6"}
Jul 01 19:37:47.847621 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.847511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerStarted","Data":"234901cfab937bad1a145560841d1512d9e6d6f6dcbf98b1747910b19617ea66"}
Jul 01 19:37:47.849020 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.848996    2578 generic.go:356] "Generic (PLEG): container finished" podID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerID="1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88" exitCode=2
Jul 01 19:37:47.849020 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.849018    2578 generic.go:356] "Generic (PLEG): container finished" podID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerID="fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6" exitCode=2
Jul 01 19:37:47.849202 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.849060    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"
Jul 01 19:37:47.849202 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.849075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" event={"ID":"823baaa0-2b4e-49cd-8900-11ce6b54d01d","Type":"ContainerDied","Data":"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"}
Jul 01 19:37:47.849202 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.849125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" event={"ID":"823baaa0-2b4e-49cd-8900-11ce6b54d01d","Type":"ContainerDied","Data":"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"}
Jul 01 19:37:47.849202 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.849150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx" event={"ID":"823baaa0-2b4e-49cd-8900-11ce6b54d01d","Type":"ContainerDied","Data":"25e36be23eb819ed07c65480d17128fa5477fa3683710f742eaf9332b2528d6e"}
Jul 01 19:37:47.849202 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.849175    2578 scope.go:117] "RemoveContainer" containerID="1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"
Jul 01 19:37:47.857367 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.857350    2578 scope.go:117] "RemoveContainer" containerID="fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"
Jul 01 19:37:47.864440 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.864424    2578 scope.go:117] "RemoveContainer" containerID="1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"
Jul 01 19:37:47.864703 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:37:47.864685    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88\": container with ID starting with 1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88 not found: ID does not exist" containerID="1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"
Jul 01 19:37:47.864755 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.864711    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"} err="failed to get container status \"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88\": rpc error: code = NotFound desc = could not find container \"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88\": container with ID starting with 1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88 not found: ID does not exist"
Jul 01 19:37:47.864755 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.864731    2578 scope.go:117] "RemoveContainer" containerID="fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"
Jul 01 19:37:47.864997 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:37:47.864974    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6\": container with ID starting with fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6 not found: ID does not exist" containerID="fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"
Jul 01 19:37:47.865046 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.865006    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"} err="failed to get container status \"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6\": rpc error: code = NotFound desc = could not find container \"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6\": container with ID starting with fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6 not found: ID does not exist"
Jul 01 19:37:47.865046 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.865027    2578 scope.go:117] "RemoveContainer" containerID="1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"
Jul 01 19:37:47.865277 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.865254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88"} err="failed to get container status \"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88\": rpc error: code = NotFound desc = could not find container \"1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88\": container with ID starting with 1b4e772dbf4c93e640e22cfe33fcd400de495c17daff6064e002c8959a446b88 not found: ID does not exist"
Jul 01 19:37:47.865328 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.865279    2578 scope.go:117] "RemoveContainer" containerID="fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"
Jul 01 19:37:47.865503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.865483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6"} err="failed to get container status \"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6\": rpc error: code = NotFound desc = could not find container \"fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6\": container with ID starting with fd2d856f3ca3bb117a2677283f1d68cbd3360b5b00b5f9870e236b87c43984a6 not found: ID does not exist"
Jul 01 19:37:47.880943 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.880915    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"]
Jul 01 19:37:47.884740 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:47.884718    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-gxcmx"]
Jul 01 19:37:48.506166 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:48.506129    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:37:49.444477 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:49.444445    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" path="/var/lib/kubelet/pods/823baaa0-2b4e-49cd-8900-11ce6b54d01d/volumes"
Jul 01 19:37:51.867748 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:51.867716    2578 generic.go:356] "Generic (PLEG): container finished" podID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerID="9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564" exitCode=0
Jul 01 19:37:51.868072 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:51.867788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerDied","Data":"9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564"}
Jul 01 19:37:53.506133 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:53.506094    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:37:54.439906 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:54.439862    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:37:54.440815 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:54.440779    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:37:58.505968 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:58.505918    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:37:58.509876 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:37:58.506077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:38:03.505883 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:03.505844    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:38:04.439963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:04.439927    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:38:04.440704 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:04.440675    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:07.932350 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:07.932315    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerID="a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6" exitCode=0
Jul 01 19:38:07.932769 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:07.932384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerDied","Data":"a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6"}
Jul 01 19:38:08.505981 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:08.505934    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:38:13.506006 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:13.505958    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 01 19:38:14.439952 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:14.439903    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 01 19:38:14.440151 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:14.440056    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:38:14.440848 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:14.440811    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:38:14.440969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:14.440932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:38:17.145694 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.145671    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:38:17.180578 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.180551    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vr5d\" (UniqueName: \"kubernetes.io/projected/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-api-access-9vr5d\") pod \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") "
Jul 01 19:38:17.180771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.180609    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kserve-provision-location\") pod \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") "
Jul 01 19:38:17.180771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.180651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-cabundle-cert\") pod \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") "
Jul 01 19:38:17.180771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.180689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-rbac-proxy-sar-config\") pod \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") "
Jul 01 19:38:17.180771 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.180749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls\") pod \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\" (UID: \"94a7be8e-e234-4a30-9613-1f4f7c7cc378\") "
Jul 01 19:38:17.181023 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.180937    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94a7be8e-e234-4a30-9613-1f4f7c7cc378" (UID: "94a7be8e-e234-4a30-9613-1f4f7c7cc378"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:38:17.181268 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.181243    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94a7be8e-e234-4a30-9613-1f4f7c7cc378" (UID: "94a7be8e-e234-4a30-9613-1f4f7c7cc378"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:38:17.181381 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.181345    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94a7be8e-e234-4a30-9613-1f4f7c7cc378" (UID: "94a7be8e-e234-4a30-9613-1f4f7c7cc378"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:38:17.183303 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.183269    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94a7be8e-e234-4a30-9613-1f4f7c7cc378" (UID: "94a7be8e-e234-4a30-9613-1f4f7c7cc378"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:38:17.183303 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.183275    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-api-access-9vr5d" (OuterVolumeSpecName: "kube-api-access-9vr5d") pod "94a7be8e-e234-4a30-9613-1f4f7c7cc378" (UID: "94a7be8e-e234-4a30-9613-1f4f7c7cc378"). InnerVolumeSpecName "kube-api-access-9vr5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:38:17.282035 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.281940    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94a7be8e-e234-4a30-9613-1f4f7c7cc378-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:38:17.282035 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.281976    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vr5d\" (UniqueName: \"kubernetes.io/projected/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-api-access-9vr5d\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:38:17.282035 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.281992    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:38:17.282035 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.282006    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:38:17.282035 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.282026    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94a7be8e-e234-4a30-9613-1f4f7c7cc378-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:38:17.976048 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.976018    2578 generic.go:356] "Generic (PLEG): container finished" podID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerID="ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1" exitCode=0
Jul 01 19:38:17.976222 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.976058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerDied","Data":"ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1"}
Jul 01 19:38:17.976222 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.976083    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"
Jul 01 19:38:17.976222 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.976108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9" event={"ID":"94a7be8e-e234-4a30-9613-1f4f7c7cc378","Type":"ContainerDied","Data":"686d60ec40f5ca89c918a0c73c6d503cc673b4b2e83a822de8e2107977b79ca3"}
Jul 01 19:38:17.976222 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.976138    2578 scope.go:117] "RemoveContainer" containerID="ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1"
Jul 01 19:38:17.983992 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.983975    2578 scope.go:117] "RemoveContainer" containerID="a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f"
Jul 01 19:38:17.991049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.991033    2578 scope.go:117] "RemoveContainer" containerID="9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564"
Jul 01 19:38:17.994050 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.994027    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"]
Jul 01 19:38:17.998720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.998580    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-4csn9"]
Jul 01 19:38:17.998720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:17.998707    2578 scope.go:117] "RemoveContainer" containerID="ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee"
Jul 01 19:38:18.005827 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.005811    2578 scope.go:117] "RemoveContainer" containerID="ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1"
Jul 01 19:38:18.006073 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:38:18.006056    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1\": container with ID starting with ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1 not found: ID does not exist" containerID="ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1"
Jul 01 19:38:18.006121 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006082    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1"} err="failed to get container status \"ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1\": rpc error: code = NotFound desc = could not find container \"ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1\": container with ID starting with ee8ffb6c3bf1af5474f34765186ef1ebc61285424ad3aea9c6c072b9834278d1 not found: ID does not exist"
Jul 01 19:38:18.006121 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006101    2578 scope.go:117] "RemoveContainer" containerID="a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f"
Jul 01 19:38:18.006345 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:38:18.006326    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f\": container with ID starting with a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f not found: ID does not exist" containerID="a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f"
Jul 01 19:38:18.006386 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006352    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f"} err="failed to get container status \"a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f\": rpc error: code = NotFound desc = could not find container \"a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f\": container with ID starting with a1214fa47da51ac4602ba2b22ec68a15bdea5838f19b747063bdc5155bc9b11f not found: ID does not exist"
Jul 01 19:38:18.006386 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006369    2578 scope.go:117] "RemoveContainer" containerID="9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564"
Jul 01 19:38:18.006612 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:38:18.006596    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564\": container with ID starting with 9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564 not found: ID does not exist" containerID="9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564"
Jul 01 19:38:18.006720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006617    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564"} err="failed to get container status \"9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564\": rpc error: code = NotFound desc = could not find container \"9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564\": container with ID starting with 9822867d940eefa7ea8aee97ea1b9fd240e5ed4b22962332a677b030175f9564 not found: ID does not exist"
Jul 01 19:38:18.006720 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006657    2578 scope.go:117] "RemoveContainer" containerID="ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee"
Jul 01 19:38:18.006859 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:38:18.006840    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee\": container with ID starting with ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee not found: ID does not exist" containerID="ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee"
Jul 01 19:38:18.006906 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:18.006866    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee"} err="failed to get container status \"ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee\": rpc error: code = NotFound desc = could not find container \"ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee\": container with ID starting with ceae9809ced7b5e97add3dbb22c567aee9ecc84656c2b107cac09e67c1292aee not found: ID does not exist"
Jul 01 19:38:19.444978 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:38:19.444936    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" path="/var/lib/kubelet/pods/94a7be8e-e234-4a30-9613-1f4f7c7cc378/volumes"
Jul 01 19:40:30.517485 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:30.517452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerStarted","Data":"3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d"}
Jul 01 19:40:30.518007 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:30.517507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerStarted","Data":"0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144"}
Jul 01 19:40:30.518007 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:30.517662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:40:30.518007 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:30.517706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:40:30.518996 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:30.518972    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:30.548845 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:30.548789    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" podStartSLOduration=2.748519925 podStartE2EDuration="2m44.548772124s" totalImagesPullingTime="2m21.508640274s" totalInitContainerRuntime="20.291611925s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:40:30.54690796 +0000 UTC m=+1095.647796231" watchObservedRunningTime="2026-07-01 19:40:30.548772124 +0000 UTC m=+1095.649660382"
Jul 01 19:40:31.522118 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:31.522083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:36.527571 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:36.527537    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:40:36.528123 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:36.528092    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 19:40:46.528586 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:46.528550    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:40:47.114073 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:47.114042    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"]
Jul 01 19:40:47.114279 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:47.114257    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container" containerID="cri-o://0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144" gracePeriod=30
Jul 01 19:40:47.114353 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:47.114301    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kube-rbac-proxy" containerID="cri-o://3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d" gracePeriod=30
Jul 01 19:40:47.586955 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:47.586923    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerID="3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d" exitCode=2
Jul 01 19:40:47.587330 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:47.586991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerDied","Data":"3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d"}
Jul 01 19:40:48.084218 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.084190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:40:48.125542 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.125469    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-rbac-proxy-sar-config\") pod \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") "
Jul 01 19:40:48.125542 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.125502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-proxy-tls\") pod \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") "
Jul 01 19:40:48.125542 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.125519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-cabundle-cert\") pod \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") "
Jul 01 19:40:48.125779 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.125571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx4q7\" (UniqueName: \"kubernetes.io/projected/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-api-access-nx4q7\") pod \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") "
Jul 01 19:40:48.125779 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.125597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kserve-provision-location\") pod \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\" (UID: \"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9\") "
Jul 01 19:40:48.126001 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.125980    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" (UID: "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:48.126119 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.126089    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" (UID: "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:40:48.127668 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.127645    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-api-access-nx4q7" (OuterVolumeSpecName: "kube-api-access-nx4q7") pod "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" (UID: "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9"). InnerVolumeSpecName "kube-api-access-nx4q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:40:48.127791 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.127774    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" (UID: "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:40:48.130074 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.130051    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" (UID: "3c90f011-879c-48e4-b3cd-ff2c43b8c8a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:40:48.226181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.226138    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx4q7\" (UniqueName: \"kubernetes.io/projected/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-api-access-nx4q7\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:40:48.226181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.226174    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:40:48.226181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.226187    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:40:48.226415 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.226199    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:40:48.226415 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.226211    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:40:48.592741 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.592708    2578 generic.go:356] "Generic (PLEG): container finished" podID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerID="0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144" exitCode=0
Jul 01 19:40:48.593232 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.592780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"
Jul 01 19:40:48.593232 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.592797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerDied","Data":"0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144"}
Jul 01 19:40:48.593232 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.592855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh" event={"ID":"3c90f011-879c-48e4-b3cd-ff2c43b8c8a9","Type":"ContainerDied","Data":"234901cfab937bad1a145560841d1512d9e6d6f6dcbf98b1747910b19617ea66"}
Jul 01 19:40:48.593232 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.592883    2578 scope.go:117] "RemoveContainer" containerID="3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d"
Jul 01 19:40:48.606602 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.606581    2578 scope.go:117] "RemoveContainer" containerID="0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144"
Jul 01 19:40:48.614105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.614085    2578 scope.go:117] "RemoveContainer" containerID="a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6"
Jul 01 19:40:48.619161 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.619138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"]
Jul 01 19:40:48.622049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622026    2578 scope.go:117] "RemoveContainer" containerID="3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d"
Jul 01 19:40:48.622344 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:40:48.622325    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d\": container with ID starting with 3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d not found: ID does not exist" containerID="3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d"
Jul 01 19:40:48.622392 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622353    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d"} err="failed to get container status \"3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d\": rpc error: code = NotFound desc = could not find container \"3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d\": container with ID starting with 3aca244e782c4c1de367bce7408b6c54491ec8f052ffc458000f2d2a46673e6d not found: ID does not exist"
Jul 01 19:40:48.622392 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622372    2578 scope.go:117] "RemoveContainer" containerID="0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144"
Jul 01 19:40:48.622575 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:40:48.622560    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144\": container with ID starting with 0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144 not found: ID does not exist" containerID="0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144"
Jul 01 19:40:48.622616 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622579    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144"} err="failed to get container status \"0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144\": rpc error: code = NotFound desc = could not find container \"0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144\": container with ID starting with 0a124597fb7096cb551b361a5f1717735a24f96aa612d68809bc142b5c6a8144 not found: ID does not exist"
Jul 01 19:40:48.622616 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622594    2578 scope.go:117] "RemoveContainer" containerID="a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6"
Jul 01 19:40:48.622731 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622671    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-cxgmh"]
Jul 01 19:40:48.622871 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:40:48.622855    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6\": container with ID starting with a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6 not found: ID does not exist" containerID="a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6"
Jul 01 19:40:48.622909 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:48.622877    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6"} err="failed to get container status \"a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6\": rpc error: code = NotFound desc = could not find container \"a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6\": container with ID starting with a400a9ac9ad15717fef06e5180220189327556306132418661e3f5a8fea39fe6 not found: ID does not exist"
Jul 01 19:40:49.444260 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:49.444231    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" path="/var/lib/kubelet/pods/3c90f011-879c-48e4-b3cd-ff2c43b8c8a9/volumes"
Jul 01 19:40:52.371094 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371062    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"]
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kube-rbac-proxy"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371334    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kube-rbac-proxy"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371354    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371366    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="storage-initializer"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371374    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="storage-initializer"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371385    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371392    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371402    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371412    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371426    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371433    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371442    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="storage-initializer"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371446    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="storage-initializer"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371452    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kserve-container"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371457    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kserve-container"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371465    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kube-rbac-proxy"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371470    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kube-rbac-proxy"
Jul 01 19:40:52.371503 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371512    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kube-rbac-proxy"
Jul 01 19:40:52.372238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371520    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kserve-container"
Jul 01 19:40:52.372238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371529    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kube-rbac-proxy"
Jul 01 19:40:52.372238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371535    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="kserve-container"
Jul 01 19:40:52.372238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c90f011-879c-48e4-b3cd-ff2c43b8c8a9" containerName="kube-rbac-proxy"
Jul 01 19:40:52.372238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371545    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="823baaa0-2b4e-49cd-8900-11ce6b54d01d" containerName="kserve-container"
Jul 01 19:40:52.372238 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.371550    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a7be8e-e234-4a30-9613-1f4f7c7cc378" containerName="agent"
Jul 01 19:40:52.406954 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.406924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"]
Jul 01 19:40:52.407101 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.407033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.409582 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.409559    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:40:52.409729 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.409558    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:40:52.409820 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.409742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zft66\""
Jul 01 19:40:52.409820 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.409776    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 01 19:40:52.409935 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.409850    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:40:52.409995 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.409937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 01 19:40:52.410479 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.410465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:40:52.452133 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.452101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.452133 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.452137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbt6d\" (UniqueName: \"kubernetes.io/projected/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-api-access-tbt6d\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.452402 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.452163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e923983-991d-4bd7-8217-4c18f2657cd1-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.452402 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.452196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.452402 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.452217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e923983-991d-4bd7-8217-4c18f2657cd1-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.552950 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.552913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.552950 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.552951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbt6d\" (UniqueName: \"kubernetes.io/projected/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-api-access-tbt6d\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.553193 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.552977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e923983-991d-4bd7-8217-4c18f2657cd1-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.553193 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.553002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.553193 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.553019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e923983-991d-4bd7-8217-4c18f2657cd1-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.553438 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.553421    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e923983-991d-4bd7-8217-4c18f2657cd1-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.553728 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.553709    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.553781 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.553723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.555579 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.555558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e923983-991d-4bd7-8217-4c18f2657cd1-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.561182 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.561159    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbt6d\" (UniqueName: \"kubernetes.io/projected/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-api-access-tbt6d\") pod \"isvc-autogluon-v2-predictor-67b849d456-h95pv\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.718006 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.717968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:52.871469 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:52.871439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"]
Jul 01 19:40:52.874482 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:40:52.874460    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e923983_991d_4bd7_8217_4c18f2657cd1.slice/crio-a67940bc83a447fd97f67c8a376808931e234efa9e966cb1536aac828bc308e9 WatchSource:0}: Error finding container a67940bc83a447fd97f67c8a376808931e234efa9e966cb1536aac828bc308e9: Status 404 returned error can't find the container with id a67940bc83a447fd97f67c8a376808931e234efa9e966cb1536aac828bc308e9
Jul 01 19:40:53.612404 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:53.612369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerStarted","Data":"3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252"}
Jul 01 19:40:53.612914 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:53.612442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerStarted","Data":"a67940bc83a447fd97f67c8a376808931e234efa9e966cb1536aac828bc308e9"}
Jul 01 19:40:58.632213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:58.632178    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerID="3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252" exitCode=0
Jul 01 19:40:58.632663 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:58.632229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerDied","Data":"3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252"}
Jul 01 19:40:59.637822 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:59.637787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerStarted","Data":"4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02"}
Jul 01 19:40:59.638231 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:59.637858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerStarted","Data":"1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d"}
Jul 01 19:40:59.638231 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:59.638053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:40:59.659079 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:40:59.659023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" podStartSLOduration=2.785470063 podStartE2EDuration="7.65900668s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.873536617s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:40:59.65766703 +0000 UTC m=+1124.758555286" watchObservedRunningTime="2026-07-01 19:40:59.65900668 +0000 UTC m=+1124.759894939"
Jul 01 19:41:00.642274 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:41:00.642242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:41:06.651931 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:41:06.651902    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:42:37.444953 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:37.444922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:42:42.618594 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:42.618562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"]
Jul 01 19:42:42.619055 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:42.618804    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kserve-container" containerID="cri-o://1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d" gracePeriod=30
Jul 01 19:42:42.619055 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:42.618853    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kube-rbac-proxy" containerID="cri-o://4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02" gracePeriod=30
Jul 01 19:42:43.014068 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.014038    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerID="4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02" exitCode=2
Jul 01 19:42:43.014240 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.014103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerDied","Data":"4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02"}
Jul 01 19:42:43.463074 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.463045    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:42:43.576755 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.576680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-rbac-proxy-sar-config\") pod \"3e923983-991d-4bd7-8217-4c18f2657cd1\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") "
Jul 01 19:42:43.576755 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.576733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-cabundle-cert\") pod \"3e923983-991d-4bd7-8217-4c18f2657cd1\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") "
Jul 01 19:42:43.576951 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.576764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbt6d\" (UniqueName: \"kubernetes.io/projected/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-api-access-tbt6d\") pod \"3e923983-991d-4bd7-8217-4c18f2657cd1\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") "
Jul 01 19:42:43.576951 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.576818    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e923983-991d-4bd7-8217-4c18f2657cd1-proxy-tls\") pod \"3e923983-991d-4bd7-8217-4c18f2657cd1\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") "
Jul 01 19:42:43.576951 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.576857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e923983-991d-4bd7-8217-4c18f2657cd1-kserve-provision-location\") pod \"3e923983-991d-4bd7-8217-4c18f2657cd1\" (UID: \"3e923983-991d-4bd7-8217-4c18f2657cd1\") "
Jul 01 19:42:43.577110 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.577085    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e923983-991d-4bd7-8217-4c18f2657cd1" (UID: "3e923983-991d-4bd7-8217-4c18f2657cd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:43.577175 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.577148    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e923983-991d-4bd7-8217-4c18f2657cd1" (UID: "3e923983-991d-4bd7-8217-4c18f2657cd1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:43.578873 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.578844    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e923983-991d-4bd7-8217-4c18f2657cd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e923983-991d-4bd7-8217-4c18f2657cd1" (UID: "3e923983-991d-4bd7-8217-4c18f2657cd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:43.579215 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.579197    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-api-access-tbt6d" (OuterVolumeSpecName: "kube-api-access-tbt6d") pod "3e923983-991d-4bd7-8217-4c18f2657cd1" (UID: "3e923983-991d-4bd7-8217-4c18f2657cd1"). InnerVolumeSpecName "kube-api-access-tbt6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:42:43.582454 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.582403    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e923983-991d-4bd7-8217-4c18f2657cd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e923983-991d-4bd7-8217-4c18f2657cd1" (UID: "3e923983-991d-4bd7-8217-4c18f2657cd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:42:43.677768 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.677730    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:42:43.677768 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.677763    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e923983-991d-4bd7-8217-4c18f2657cd1-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:42:43.677768 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.677773    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbt6d\" (UniqueName: \"kubernetes.io/projected/3e923983-991d-4bd7-8217-4c18f2657cd1-kube-api-access-tbt6d\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:42:43.678201 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.677783    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e923983-991d-4bd7-8217-4c18f2657cd1-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:42:43.678201 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:43.677792    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e923983-991d-4bd7-8217-4c18f2657cd1-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:42:44.019308 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.019278    2578 generic.go:356] "Generic (PLEG): container finished" podID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerID="1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d" exitCode=0
Jul 01 19:42:44.019464 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.019338    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"
Jul 01 19:42:44.019464 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.019354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerDied","Data":"1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d"}
Jul 01 19:42:44.019464 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.019419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv" event={"ID":"3e923983-991d-4bd7-8217-4c18f2657cd1","Type":"ContainerDied","Data":"a67940bc83a447fd97f67c8a376808931e234efa9e966cb1536aac828bc308e9"}
Jul 01 19:42:44.019464 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.019457    2578 scope.go:117] "RemoveContainer" containerID="4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02"
Jul 01 19:42:44.027894 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.027875    2578 scope.go:117] "RemoveContainer" containerID="1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d"
Jul 01 19:42:44.035025 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.035004    2578 scope.go:117] "RemoveContainer" containerID="3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252"
Jul 01 19:42:44.040974 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.040949    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"]
Jul 01 19:42:44.042732 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.042712    2578 scope.go:117] "RemoveContainer" containerID="4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02"
Jul 01 19:42:44.043016 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:42:44.042995    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02\": container with ID starting with 4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02 not found: ID does not exist" containerID="4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02"
Jul 01 19:42:44.043065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.043026    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02"} err="failed to get container status \"4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02\": rpc error: code = NotFound desc = could not find container \"4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02\": container with ID starting with 4a3b2fdfd4b31ea64a5300928bcbe9fa3a5bc102a5c11bffb963f587babd5b02 not found: ID does not exist"
Jul 01 19:42:44.043065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.043047    2578 scope.go:117] "RemoveContainer" containerID="1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d"
Jul 01 19:42:44.043137 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.043080    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-h95pv"]
Jul 01 19:42:44.043310 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:42:44.043293    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d\": container with ID starting with 1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d not found: ID does not exist" containerID="1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d"
Jul 01 19:42:44.043352 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.043317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d"} err="failed to get container status \"1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d\": rpc error: code = NotFound desc = could not find container \"1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d\": container with ID starting with 1bff126f753a4819f669188239be1f3cf819e7996be7c68f2e8c14bec97a746d not found: ID does not exist"
Jul 01 19:42:44.043352 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.043333    2578 scope.go:117] "RemoveContainer" containerID="3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252"
Jul 01 19:42:44.043572 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:42:44.043558    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252\": container with ID starting with 3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252 not found: ID does not exist" containerID="3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252"
Jul 01 19:42:44.043612 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:44.043575    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252"} err="failed to get container status \"3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252\": rpc error: code = NotFound desc = could not find container \"3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252\": container with ID starting with 3941c1a4364a9ca05c021ba767494d6324f41978e8db51c0e2920b0705910252 not found: ID does not exist"
Jul 01 19:42:45.443650 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:45.443614    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" path="/var/lib/kubelet/pods/3e923983-991d-4bd7-8217-4c18f2657cd1/volumes"
Jul 01 19:42:47.709314 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709278    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"]
Jul 01 19:42:47.709793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709649    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kserve-container"
Jul 01 19:42:47.709793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709668    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kserve-container"
Jul 01 19:42:47.709793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="storage-initializer"
Jul 01 19:42:47.709793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709697    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="storage-initializer"
Jul 01 19:42:47.709793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709716    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kube-rbac-proxy"
Jul 01 19:42:47.709793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709724    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kube-rbac-proxy"
Jul 01 19:42:47.710105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709814    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kube-rbac-proxy"
Jul 01 19:42:47.710105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.709828    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e923983-991d-4bd7-8217-4c18f2657cd1" containerName="kserve-container"
Jul 01 19:42:47.713008 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.712987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.715422 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.715386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zft66\""
Jul 01 19:42:47.715545 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.715440    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:42:47.715545 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.715535    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 01 19:42:47.716370 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.716351    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:42:47.716501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.716391    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 01 19:42:47.716501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.716355    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:42:47.716501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.716445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:42:47.728100 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.728076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"]
Jul 01 19:42:47.809550 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.809468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jw6\" (UniqueName: \"kubernetes.io/projected/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-api-access-p4jw6\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.809550 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.809520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.809550 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.809543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.809806 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.809583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.809806 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.809613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.910919 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.910883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jw6\" (UniqueName: \"kubernetes.io/projected/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-api-access-p4jw6\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.910929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.910955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.910979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911049 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.911004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911422 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.911401    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911563 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.911545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.911612 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.911594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.913395 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.913379    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:47.919275 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:47.919250    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jw6\" (UniqueName: \"kubernetes.io/projected/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-api-access-p4jw6\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:48.028979 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:48.028940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:42:48.155254 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:48.155220    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"]
Jul 01 19:42:48.158355 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:42:48.158326    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e8d72b_e963_48fd_8ac4_2c1d10824ef4.slice/crio-770f7f3600ca3fdee17045a2ebe0b78b8e491f978d03455038fbb8f8a0d7d9ee WatchSource:0}: Error finding container 770f7f3600ca3fdee17045a2ebe0b78b8e491f978d03455038fbb8f8a0d7d9ee: Status 404 returned error can't find the container with id 770f7f3600ca3fdee17045a2ebe0b78b8e491f978d03455038fbb8f8a0d7d9ee
Jul 01 19:42:49.040423 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:49.040385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerStarted","Data":"a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d"}
Jul 01 19:42:49.040825 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:42:49.040457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerStarted","Data":"770f7f3600ca3fdee17045a2ebe0b78b8e491f978d03455038fbb8f8a0d7d9ee"}
Jul 01 19:43:09.117014 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:09.116983    2578 generic.go:356] "Generic (PLEG): container finished" podID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerID="a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d" exitCode=0
Jul 01 19:43:09.117475 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:09.117054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerDied","Data":"a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d"}
Jul 01 19:43:10.123574 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:10.123537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerStarted","Data":"6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93"}
Jul 01 19:43:10.123966 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:10.123597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerStarted","Data":"64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee"}
Jul 01 19:43:10.123966 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:10.123808    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:43:10.143417 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:10.143341    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" podStartSLOduration=3.1167323749999998 podStartE2EDuration="23.143323293s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.026590918s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:10.141939867 +0000 UTC m=+1255.242828150" watchObservedRunningTime="2026-07-01 19:43:10.143323293 +0000 UTC m=+1255.244211594"
Jul 01 19:43:11.127078 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:11.127037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:43:17.141534 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:43:17.141508    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:44:48.444480 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:48.444387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:44:57.839009 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:57.838974    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"]
Jul 01 19:44:57.839572 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:57.839200    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kserve-container" containerID="cri-o://64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee" gracePeriod=30
Jul 01 19:44:57.839572 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:57.839238    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kube-rbac-proxy" containerID="cri-o://6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93" gracePeriod=30
Jul 01 19:44:58.440877 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.440833    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 19:44:58.514061 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.514023    2578 generic.go:356] "Generic (PLEG): container finished" podID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerID="6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93" exitCode=2
Jul 01 19:44:58.514211 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.514093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerDied","Data":"6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93"}
Jul 01 19:44:58.690864 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.690841    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:44:58.763969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.763876    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-proxy-tls\") pod \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") "
Jul 01 19:44:58.763969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.763928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-cabundle-cert\") pod \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") "
Jul 01 19:44:58.763969 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.763959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4jw6\" (UniqueName: \"kubernetes.io/projected/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-api-access-p4jw6\") pod \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") "
Jul 01 19:44:58.764237 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.764011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-rbac-proxy-sar-config\") pod \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") "
Jul 01 19:44:58.764237 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.764042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kserve-provision-location\") pod \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\" (UID: \"58e8d72b-e963-48fd-8ac4-2c1d10824ef4\") "
Jul 01 19:44:58.764372 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.764342    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58e8d72b-e963-48fd-8ac4-2c1d10824ef4" (UID: "58e8d72b-e963-48fd-8ac4-2c1d10824ef4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:58.764372 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.764357    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58e8d72b-e963-48fd-8ac4-2c1d10824ef4" (UID: "58e8d72b-e963-48fd-8ac4-2c1d10824ef4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:44:58.766080 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.766055    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-api-access-p4jw6" (OuterVolumeSpecName: "kube-api-access-p4jw6") pod "58e8d72b-e963-48fd-8ac4-2c1d10824ef4" (UID: "58e8d72b-e963-48fd-8ac4-2c1d10824ef4"). InnerVolumeSpecName "kube-api-access-p4jw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:44:58.766160 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.766093    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58e8d72b-e963-48fd-8ac4-2c1d10824ef4" (UID: "58e8d72b-e963-48fd-8ac4-2c1d10824ef4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:44:58.768984 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.768964    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58e8d72b-e963-48fd-8ac4-2c1d10824ef4" (UID: "58e8d72b-e963-48fd-8ac4-2c1d10824ef4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:44:58.865398 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.865364    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:44:58.865398 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.865393    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:44:58.865815 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.865405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4jw6\" (UniqueName: \"kubernetes.io/projected/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-api-access-p4jw6\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:44:58.865815 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.865418    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:44:58.865815 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:58.865429    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58e8d72b-e963-48fd-8ac4-2c1d10824ef4-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:44:59.519703 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.519672    2578 generic.go:356] "Generic (PLEG): container finished" podID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerID="64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee" exitCode=0
Jul 01 19:44:59.519854 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.519710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerDied","Data":"64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee"}
Jul 01 19:44:59.519854 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.519725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"
Jul 01 19:44:59.519854 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.519762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd" event={"ID":"58e8d72b-e963-48fd-8ac4-2c1d10824ef4","Type":"ContainerDied","Data":"770f7f3600ca3fdee17045a2ebe0b78b8e491f978d03455038fbb8f8a0d7d9ee"}
Jul 01 19:44:59.519854 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.519788    2578 scope.go:117] "RemoveContainer" containerID="6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93"
Jul 01 19:44:59.527675 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.527649    2578 scope.go:117] "RemoveContainer" containerID="64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee"
Jul 01 19:44:59.535208 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.535187    2578 scope.go:117] "RemoveContainer" containerID="a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d"
Jul 01 19:44:59.543531 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.543500    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"]
Jul 01 19:44:59.546898 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.546877    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-l42fd"]
Jul 01 19:44:59.547235 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.547223    2578 scope.go:117] "RemoveContainer" containerID="6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93"
Jul 01 19:44:59.547500 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:44:59.547482    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93\": container with ID starting with 6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93 not found: ID does not exist" containerID="6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93"
Jul 01 19:44:59.547555 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.547507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93"} err="failed to get container status \"6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93\": rpc error: code = NotFound desc = could not find container \"6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93\": container with ID starting with 6ad60b14faf0a196a0bd111fb3b9c551514ef025ece4dd2049e4d3d28db7df93 not found: ID does not exist"
Jul 01 19:44:59.547555 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.547524    2578 scope.go:117] "RemoveContainer" containerID="64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee"
Jul 01 19:44:59.547771 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:44:59.547757    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee\": container with ID starting with 64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee not found: ID does not exist" containerID="64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee"
Jul 01 19:44:59.547824 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.547774    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee"} err="failed to get container status \"64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee\": rpc error: code = NotFound desc = could not find container \"64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee\": container with ID starting with 64cb4d58946eeae5a71fc1da9f6d6911a99b125c8fc7bbf65f21c31dbb1f85ee not found: ID does not exist"
Jul 01 19:44:59.547824 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.547786    2578 scope.go:117] "RemoveContainer" containerID="a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d"
Jul 01 19:44:59.547969 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:44:59.547955    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d\": container with ID starting with a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d not found: ID does not exist" containerID="a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d"
Jul 01 19:44:59.548001 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:44:59.547973    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d"} err="failed to get container status \"a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d\": rpc error: code = NotFound desc = could not find container \"a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d\": container with ID starting with a6449cafd280c1f7ee1ca34e32e2ae10eab00f5578a0c687162d07bb9fb8dc7d not found: ID does not exist"
Jul 01 19:45:01.443459 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:01.443424    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" path="/var/lib/kubelet/pods/58e8d72b-e963-48fd-8ac4-2c1d10824ef4/volumes"
Jul 01 19:45:03.010867 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.010835    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"]
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011075    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="storage-initializer"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011086    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="storage-initializer"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011101    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kserve-container"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011106    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kserve-container"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011116    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kube-rbac-proxy"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011121    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kube-rbac-proxy"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011167    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kserve-container"
Jul 01 19:45:03.011236 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.011175    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e8d72b-e963-48fd-8ac4-2c1d10824ef4" containerName="kube-rbac-proxy"
Jul 01 19:45:03.015592 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.015573    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.017983 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.017956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 01 19:45:03.018290 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.018273    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:45:03.018403 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.018387    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:45:03.018469 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.018456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zft66\""
Jul 01 19:45:03.018574 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.018562    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 01 19:45:03.019012 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.018993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:45:03.019170 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.019157    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:45:03.028414 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.028356    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"]
Jul 01 19:45:03.093772 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.093735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354d8d61-95c8-4db2-9ab6-9d8962b91360-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.093963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.093779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28hx8\" (UniqueName: \"kubernetes.io/projected/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-api-access-28hx8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.093963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.093856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.093963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.093906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.093963 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.093934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/354d8d61-95c8-4db2-9ab6-9d8962b91360-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195121 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195121 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/354d8d61-95c8-4db2-9ab6-9d8962b91360-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195333 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354d8d61-95c8-4db2-9ab6-9d8962b91360-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195333 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28hx8\" (UniqueName: \"kubernetes.io/projected/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-api-access-28hx8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195439 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195593 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195576    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354d8d61-95c8-4db2-9ab6-9d8962b91360-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195852 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195826    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.195974 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.195959    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.197616 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.197602    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/354d8d61-95c8-4db2-9ab6-9d8962b91360-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.204554 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.204536    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28hx8\" (UniqueName: \"kubernetes.io/projected/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-api-access-28hx8\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.329065 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.328969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:03.465600 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.465573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"]
Jul 01 19:45:03.468411 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:45:03.468380    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354d8d61_95c8_4db2_9ab6_9d8962b91360.slice/crio-df956934d993b8cf315ed609857ebf7e3bc7302a4dafc5932e681173ef85f116 WatchSource:0}: Error finding container df956934d993b8cf315ed609857ebf7e3bc7302a4dafc5932e681173ef85f116: Status 404 returned error can't find the container with id df956934d993b8cf315ed609857ebf7e3bc7302a4dafc5932e681173ef85f116
Jul 01 19:45:03.536576 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:03.536548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerStarted","Data":"df956934d993b8cf315ed609857ebf7e3bc7302a4dafc5932e681173ef85f116"}
Jul 01 19:45:04.541599 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:04.541562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerStarted","Data":"73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c"}
Jul 01 19:45:08.556796 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:08.556763    2578 generic.go:356] "Generic (PLEG): container finished" podID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerID="73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c" exitCode=0
Jul 01 19:45:08.557167 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:08.556835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerDied","Data":"73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c"}
Jul 01 19:45:09.562916 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:09.562885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerStarted","Data":"e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d"}
Jul 01 19:45:09.563287 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:09.562945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerStarted","Data":"27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b"}
Jul 01 19:45:09.563287 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:09.563143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:09.563287 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:09.563200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:45:09.589296 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:09.589244    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" podStartSLOduration=3.003484158 podStartE2EDuration="7.589228213s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.585744055s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:45:09.588703281 +0000 UTC m=+1374.689591539" watchObservedRunningTime="2026-07-01 19:45:09.589228213 +0000 UTC m=+1374.690116472"
Jul 01 19:45:15.573408 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:45:15.573376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:46:45.444998 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:45.444968    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:46:53.127108 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:53.127075    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"]
Jul 01 19:46:53.127572 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:53.127328    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kserve-container" containerID="cri-o://27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b" gracePeriod=30
Jul 01 19:46:53.127572 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:53.127370    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kube-rbac-proxy" containerID="cri-o://e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d" gracePeriod=30
Jul 01 19:46:53.946543 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:53.946339    2578 generic.go:356] "Generic (PLEG): container finished" podID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerID="e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d" exitCode=2
Jul 01 19:46:53.946543 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:53.946468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerDied","Data":"e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d"}
Jul 01 19:46:54.078090 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.078063    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:46:54.136921 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.136839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-rbac-proxy-sar-config\") pod \"354d8d61-95c8-4db2-9ab6-9d8962b91360\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") "
Jul 01 19:46:54.136921 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.136873    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/354d8d61-95c8-4db2-9ab6-9d8962b91360-proxy-tls\") pod \"354d8d61-95c8-4db2-9ab6-9d8962b91360\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") "
Jul 01 19:46:54.136921 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.136898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-cabundle-cert\") pod \"354d8d61-95c8-4db2-9ab6-9d8962b91360\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") "
Jul 01 19:46:54.137406 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.136924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28hx8\" (UniqueName: \"kubernetes.io/projected/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-api-access-28hx8\") pod \"354d8d61-95c8-4db2-9ab6-9d8962b91360\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") "
Jul 01 19:46:54.137406 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.136962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354d8d61-95c8-4db2-9ab6-9d8962b91360-kserve-provision-location\") pod \"354d8d61-95c8-4db2-9ab6-9d8962b91360\" (UID: \"354d8d61-95c8-4db2-9ab6-9d8962b91360\") "
Jul 01 19:46:54.137406 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.137320    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "354d8d61-95c8-4db2-9ab6-9d8962b91360" (UID: "354d8d61-95c8-4db2-9ab6-9d8962b91360"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:46:54.137406 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.137382    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "354d8d61-95c8-4db2-9ab6-9d8962b91360" (UID: "354d8d61-95c8-4db2-9ab6-9d8962b91360"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:46:54.139107 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.139076    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354d8d61-95c8-4db2-9ab6-9d8962b91360-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "354d8d61-95c8-4db2-9ab6-9d8962b91360" (UID: "354d8d61-95c8-4db2-9ab6-9d8962b91360"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:46:54.139239 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.139127    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-api-access-28hx8" (OuterVolumeSpecName: "kube-api-access-28hx8") pod "354d8d61-95c8-4db2-9ab6-9d8962b91360" (UID: "354d8d61-95c8-4db2-9ab6-9d8962b91360"). InnerVolumeSpecName "kube-api-access-28hx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:46:54.141645 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.141603    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354d8d61-95c8-4db2-9ab6-9d8962b91360-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "354d8d61-95c8-4db2-9ab6-9d8962b91360" (UID: "354d8d61-95c8-4db2-9ab6-9d8962b91360"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:46:54.238181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.238147    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:46:54.238181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.238174    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/354d8d61-95c8-4db2-9ab6-9d8962b91360-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:46:54.238181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.238184    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354d8d61-95c8-4db2-9ab6-9d8962b91360-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:46:54.238181 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.238191    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28hx8\" (UniqueName: \"kubernetes.io/projected/354d8d61-95c8-4db2-9ab6-9d8962b91360-kube-api-access-28hx8\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:46:54.238439 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.238200    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354d8d61-95c8-4db2-9ab6-9d8962b91360-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:46:54.952371 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.952339    2578 generic.go:356] "Generic (PLEG): container finished" podID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerID="27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b" exitCode=0
Jul 01 19:46:54.952549 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.952406    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"
Jul 01 19:46:54.952549 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.952427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerDied","Data":"27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b"}
Jul 01 19:46:54.952549 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.952494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z" event={"ID":"354d8d61-95c8-4db2-9ab6-9d8962b91360","Type":"ContainerDied","Data":"df956934d993b8cf315ed609857ebf7e3bc7302a4dafc5932e681173ef85f116"}
Jul 01 19:46:54.952549 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.952521    2578 scope.go:117] "RemoveContainer" containerID="e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d"
Jul 01 19:46:54.960734 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.960713    2578 scope.go:117] "RemoveContainer" containerID="27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b"
Jul 01 19:46:54.967960 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.967939    2578 scope.go:117] "RemoveContainer" containerID="73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c"
Jul 01 19:46:54.975897 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.975813    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"]
Jul 01 19:46:54.975897 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.975854    2578 scope.go:117] "RemoveContainer" containerID="e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d"
Jul 01 19:46:54.976120 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:46:54.976101    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d\": container with ID starting with e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d not found: ID does not exist" containerID="e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d"
Jul 01 19:46:54.976172 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.976127    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d"} err="failed to get container status \"e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d\": rpc error: code = NotFound desc = could not find container \"e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d\": container with ID starting with e852f5dc7550e20f1c50835c06698bc21f86a3079efb4ccd028a7b236138ac7d not found: ID does not exist"
Jul 01 19:46:54.976172 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.976149    2578 scope.go:117] "RemoveContainer" containerID="27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b"
Jul 01 19:46:54.976380 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:46:54.976363    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b\": container with ID starting with 27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b not found: ID does not exist" containerID="27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b"
Jul 01 19:46:54.976439 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.976384    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b"} err="failed to get container status \"27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b\": rpc error: code = NotFound desc = could not find container \"27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b\": container with ID starting with 27c87c36d2c602fa0081d4c278df4bc52a9a8227b7d8e4c84dc1e7db9da2d21b not found: ID does not exist"
Jul 01 19:46:54.976439 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.976399    2578 scope.go:117] "RemoveContainer" containerID="73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c"
Jul 01 19:46:54.976616 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:46:54.976600    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c\": container with ID starting with 73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c not found: ID does not exist" containerID="73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c"
Jul 01 19:46:54.976699 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.976619    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c"} err="failed to get container status \"73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c\": rpc error: code = NotFound desc = could not find container \"73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c\": container with ID starting with 73c206b9b54744e9c73ed10cc35393aae12d9f6c509e8a2f40fa4d119518d07c not found: ID does not exist"
Jul 01 19:46:54.982677 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:54.982651    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-8vk4z"]
Jul 01 19:46:55.443424 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:55.443395    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" path="/var/lib/kubelet/pods/354d8d61-95c8-4db2-9ab6-9d8962b91360/volumes"
Jul 01 19:46:58.317395 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317362    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"]
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317626    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kube-rbac-proxy"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317653    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kube-rbac-proxy"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317664    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="storage-initializer"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317670    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="storage-initializer"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317686    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kserve-container"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317692    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kserve-container"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317732    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kube-rbac-proxy"
Jul 01 19:46:58.317793 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.317740    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="354d8d61-95c8-4db2-9ab6-9d8962b91360" containerName="kserve-container"
Jul 01 19:46:58.322578 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.322561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.326058 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.326033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 01 19:46:58.326525 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.326510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zft66\""
Jul 01 19:46:58.326960 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.326943    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:46:58.327052 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.326985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:46:58.327105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.327057    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:46:58.327105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.327092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 01 19:46:58.327204 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.327093    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:46:58.337813 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.337788    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"]
Jul 01 19:46:58.366734 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.366694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.366913 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.366741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.366913 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.366779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57ph\" (UniqueName: \"kubernetes.io/projected/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-api-access-s57ph\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.366913 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.366814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.366913 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.366868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.468269 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.468233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.468462 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.468276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.468462 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.468336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.468462 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.468364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.468462 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.468398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57ph\" (UniqueName: \"kubernetes.io/projected/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-api-access-s57ph\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.468704 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:46:58.468495    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 01 19:46:58.468704 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:46:58.468557    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls podName:d5fb7d93-961f-4700-afac-ec4de2c5c9e4 nodeName:}" failed. No retries permitted until 2026-07-01 19:46:58.968540982 +0000 UTC m=+1484.069429222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" (UID: "d5fb7d93-961f-4700-afac-ec4de2c5c9e4") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 01 19:46:58.468866 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.468785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.469096 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.469074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.469131 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.469074    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.478288 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.478262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57ph\" (UniqueName: \"kubernetes.io/projected/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-api-access-s57ph\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.972724 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.972692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:58.975090 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:58.975066    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:59.233413 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:59.233327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:46:59.363912 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:59.363844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"]
Jul 01 19:46:59.366403 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:46:59.366375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fb7d93_961f_4700_afac_ec4de2c5c9e4.slice/crio-c0701326e11933877a3b671cab6911a0eec558b66cab76892fe05e00bcc71240 WatchSource:0}: Error finding container c0701326e11933877a3b671cab6911a0eec558b66cab76892fe05e00bcc71240: Status 404 returned error can't find the container with id c0701326e11933877a3b671cab6911a0eec558b66cab76892fe05e00bcc71240
Jul 01 19:46:59.972088 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:59.972056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerStarted","Data":"b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6"}
Jul 01 19:46:59.972270 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:46:59.972116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerStarted","Data":"c0701326e11933877a3b671cab6911a0eec558b66cab76892fe05e00bcc71240"}
Jul 01 19:47:08.002858 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:08.002825    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerID="b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6" exitCode=0
Jul 01 19:47:08.003241 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:08.002874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerDied","Data":"b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6"}
Jul 01 19:47:09.008211 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:09.008176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerStarted","Data":"87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f"}
Jul 01 19:47:09.008702 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:09.008246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerStarted","Data":"8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713"}
Jul 01 19:47:09.008702 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:09.008469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:47:09.036864 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:09.036814    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" podStartSLOduration=3.307468962 podStartE2EDuration="11.036797427s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.729328465s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:47:09.034758363 +0000 UTC m=+1494.135646620" watchObservedRunningTime="2026-07-01 19:47:09.036797427 +0000 UTC m=+1494.137685685"
Jul 01 19:47:10.012350 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:10.012320    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:47:10.013248 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:10.013214    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:47:11.015538 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:11.015492    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:47:16.020652 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:16.020608    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:47:16.021206 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:16.021170    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 01 19:47:26.022499 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:26.022466    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:47:28.411976 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:28.411944    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"]
Jul 01 19:47:28.412335 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:28.412139    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container" containerID="cri-o://8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713" gracePeriod=30
Jul 01 19:47:28.412335 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:28.412183    2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kube-rbac-proxy" containerID="cri-o://87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f" gracePeriod=30
Jul 01 19:47:29.086414 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.086384    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerID="87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f" exitCode=2
Jul 01 19:47:29.086584 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.086464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerDied","Data":"87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f"}
Jul 01 19:47:29.350070 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.350042    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:47:29.505172 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505134    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-cabundle-cert\") pod \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") "
Jul 01 19:47:29.505662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kserve-provision-location\") pod \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") "
Jul 01 19:47:29.505662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505212    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s57ph\" (UniqueName: \"kubernetes.io/projected/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-api-access-s57ph\") pod \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") "
Jul 01 19:47:29.505662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls\") pod \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") "
Jul 01 19:47:29.505662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505276    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-rbac-proxy-sar-config\") pod \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\" (UID: \"d5fb7d93-961f-4700-afac-ec4de2c5c9e4\") "
Jul 01 19:47:29.505662 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505557    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5fb7d93-961f-4700-afac-ec4de2c5c9e4" (UID: "d5fb7d93-961f-4700-afac-ec4de2c5c9e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:47:29.505873 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.505735    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5fb7d93-961f-4700-afac-ec4de2c5c9e4" (UID: "d5fb7d93-961f-4700-afac-ec4de2c5c9e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:47:29.506326 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.506306    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5fb7d93-961f-4700-afac-ec4de2c5c9e4" (UID: "d5fb7d93-961f-4700-afac-ec4de2c5c9e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:47:29.507385 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.507366    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5fb7d93-961f-4700-afac-ec4de2c5c9e4" (UID: "d5fb7d93-961f-4700-afac-ec4de2c5c9e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:47:29.507466 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.507431    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-api-access-s57ph" (OuterVolumeSpecName: "kube-api-access-s57ph") pod "d5fb7d93-961f-4700-afac-ec4de2c5c9e4" (UID: "d5fb7d93-961f-4700-afac-ec4de2c5c9e4"). InnerVolumeSpecName "kube-api-access-s57ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:47:29.606811 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.606732    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:47:29.606811 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.606758    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-cabundle-cert\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:47:29.606811 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.606767    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kserve-provision-location\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:47:29.606811 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.606775    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57ph\" (UniqueName: \"kubernetes.io/projected/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-kube-api-access-s57ph\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:47:29.606811 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:29.606786    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5fb7d93-961f-4700-afac-ec4de2c5c9e4-proxy-tls\") on node \"ip-10-0-131-138.ec2.internal\" DevicePath \"\""
Jul 01 19:47:30.092926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.092890    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerID="8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713" exitCode=0
Jul 01 19:47:30.093213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.092951    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"
Jul 01 19:47:30.093213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.092974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerDied","Data":"8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713"}
Jul 01 19:47:30.093213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.093036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg" event={"ID":"d5fb7d93-961f-4700-afac-ec4de2c5c9e4","Type":"ContainerDied","Data":"c0701326e11933877a3b671cab6911a0eec558b66cab76892fe05e00bcc71240"}
Jul 01 19:47:30.093213 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.093068    2578 scope.go:117] "RemoveContainer" containerID="87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f"
Jul 01 19:47:30.101005 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.100986    2578 scope.go:117] "RemoveContainer" containerID="8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713"
Jul 01 19:47:30.108541 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.108523    2578 scope.go:117] "RemoveContainer" containerID="b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6"
Jul 01 19:47:30.116966 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.116948    2578 scope.go:117] "RemoveContainer" containerID="87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f"
Jul 01 19:47:30.117221 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:47:30.117202    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f\": container with ID starting with 87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f not found: ID does not exist" containerID="87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f"
Jul 01 19:47:30.117261 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.117233    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f"} err="failed to get container status \"87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f\": rpc error: code = NotFound desc = could not find container \"87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f\": container with ID starting with 87645de70997ffa3d9139acac146b1b84dd562358b886f5e7ecaea6ee1e04d4f not found: ID does not exist"
Jul 01 19:47:30.117261 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.117251    2578 scope.go:117] "RemoveContainer" containerID="8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713"
Jul 01 19:47:30.117466 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:47:30.117449    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713\": container with ID starting with 8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713 not found: ID does not exist" containerID="8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713"
Jul 01 19:47:30.117501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.117472    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713"} err="failed to get container status \"8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713\": rpc error: code = NotFound desc = could not find container \"8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713\": container with ID starting with 8b2af78734d8ca40aa5c84015f508374a7bd9865ffeacda2d9e429b49b04e713 not found: ID does not exist"
Jul 01 19:47:30.117501 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.117488    2578 scope.go:117] "RemoveContainer" containerID="b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6"
Jul 01 19:47:30.117723 ip-10-0-131-138 kubenswrapper[2578]: E0701 19:47:30.117709    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6\": container with ID starting with b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6 not found: ID does not exist" containerID="b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6"
Jul 01 19:47:30.117772 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.117726    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6"} err="failed to get container status \"b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6\": rpc error: code = NotFound desc = could not find container \"b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6\": container with ID starting with b100256cc6a50d774934f5d96760547e7ca56c8268a5e4a4506f8d57ae5329e6 not found: ID does not exist"
Jul 01 19:47:30.125127 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.125103    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"]
Jul 01 19:47:30.141218 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:30.141195    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-7z2sg"]
Jul 01 19:47:31.444418 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:47:31.444373    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" path="/var/lib/kubelet/pods/d5fb7d93-961f-4700-afac-ec4de2c5c9e4/volumes"
Jul 01 19:48:03.026280 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026248    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26hz8/must-gather-c2t7h"]
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026486    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="storage-initializer"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026496    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="storage-initializer"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kube-rbac-proxy"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kube-rbac-proxy"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026528    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026533    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026590    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kserve-container"
Jul 01 19:48:03.026700 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.026599    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fb7d93-961f-4700-afac-ec4de2c5c9e4" containerName="kube-rbac-proxy"
Jul 01 19:48:03.029551 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.029535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.032187 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.032166    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26hz8\"/\"openshift-service-ca.crt\""
Jul 01 19:48:03.033048 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.033029    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26hz8\"/\"default-dockercfg-9bjq5\""
Jul 01 19:48:03.033048 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.033043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26hz8\"/\"kube-root-ca.crt\""
Jul 01 19:48:03.041198 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.041178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26hz8/must-gather-c2t7h"]
Jul 01 19:48:03.053768 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.053736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1969c53e-5e4a-470e-a534-2fd34d0f2be7-must-gather-output\") pod \"must-gather-c2t7h\" (UID: \"1969c53e-5e4a-470e-a534-2fd34d0f2be7\") " pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.053900 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.053789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99h2\" (UniqueName: \"kubernetes.io/projected/1969c53e-5e4a-470e-a534-2fd34d0f2be7-kube-api-access-j99h2\") pod \"must-gather-c2t7h\" (UID: \"1969c53e-5e4a-470e-a534-2fd34d0f2be7\") " pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.155105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.155075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1969c53e-5e4a-470e-a534-2fd34d0f2be7-must-gather-output\") pod \"must-gather-c2t7h\" (UID: \"1969c53e-5e4a-470e-a534-2fd34d0f2be7\") " pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.155105 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.155108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99h2\" (UniqueName: \"kubernetes.io/projected/1969c53e-5e4a-470e-a534-2fd34d0f2be7-kube-api-access-j99h2\") pod \"must-gather-c2t7h\" (UID: \"1969c53e-5e4a-470e-a534-2fd34d0f2be7\") " pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.155433 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.155403    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1969c53e-5e4a-470e-a534-2fd34d0f2be7-must-gather-output\") pod \"must-gather-c2t7h\" (UID: \"1969c53e-5e4a-470e-a534-2fd34d0f2be7\") " pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.163835 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.163806    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99h2\" (UniqueName: \"kubernetes.io/projected/1969c53e-5e4a-470e-a534-2fd34d0f2be7-kube-api-access-j99h2\") pod \"must-gather-c2t7h\" (UID: \"1969c53e-5e4a-470e-a534-2fd34d0f2be7\") " pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.338554 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.338464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26hz8/must-gather-c2t7h"
Jul 01 19:48:03.469143 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.469103    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26hz8/must-gather-c2t7h"]
Jul 01 19:48:03.471908 ip-10-0-131-138 kubenswrapper[2578]: W0701 19:48:03.471878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1969c53e_5e4a_470e_a534_2fd34d0f2be7.slice/crio-4e18d8655d3d7aaa7e1e25a195078c9cee587e9d65f03583326e29099b35f1de WatchSource:0}: Error finding container 4e18d8655d3d7aaa7e1e25a195078c9cee587e9d65f03583326e29099b35f1de: Status 404 returned error can't find the container with id 4e18d8655d3d7aaa7e1e25a195078c9cee587e9d65f03583326e29099b35f1de
Jul 01 19:48:03.473877 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:03.473862    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:48:04.215693 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:04.215646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26hz8/must-gather-c2t7h" event={"ID":"1969c53e-5e4a-470e-a534-2fd34d0f2be7","Type":"ContainerStarted","Data":"4e18d8655d3d7aaa7e1e25a195078c9cee587e9d65f03583326e29099b35f1de"}
Jul 01 19:48:05.223338 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:05.223296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26hz8/must-gather-c2t7h" event={"ID":"1969c53e-5e4a-470e-a534-2fd34d0f2be7","Type":"ContainerStarted","Data":"1f150a5396f2bb55b42f28b5635311713424f02216fa655c1d49078de2a1aa13"}
Jul 01 19:48:05.223760 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:05.223353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26hz8/must-gather-c2t7h" event={"ID":"1969c53e-5e4a-470e-a534-2fd34d0f2be7","Type":"ContainerStarted","Data":"0c817c705a1bd61a6cb7e52b3dad8fa679e4880cff3a2b6a3007bc3d046e644b"}
Jul 01 19:48:05.239001 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:05.238951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26hz8/must-gather-c2t7h" podStartSLOduration=1.478350392 podStartE2EDuration="2.238934257s" totalImagesPullingTime="760.583865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:48:05.238294865 +0000 UTC m=+1550.339183124" watchObservedRunningTime="2026-07-01 19:48:05.238934257 +0000 UTC m=+1550.339822544"
Jul 01 19:48:05.878216 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:05.878185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pjh5j_b4e1ba6d-c522-4182-b0b9-e5b3f2cbb533/global-pull-secret-syncer/0.log"
Jul 01 19:48:05.988996 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:05.988964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zgf6j_7f59f3d7-f870-44f1-88a1-a7c1de435e6a/konnectivity-agent/0.log"
Jul 01 19:48:06.021936 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:06.021903    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-138.ec2.internal_c2846b3c69993af8a8bea6ca1985de70/haproxy/0.log"
Jul 01 19:48:10.657937 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:10.657907    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5xzb7_785951ec-d3df-4e9b-89b4-06e66ef5dfe0/node-exporter/0.log"
Jul 01 19:48:10.687505 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:10.687475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5xzb7_785951ec-d3df-4e9b-89b4-06e66ef5dfe0/kube-rbac-proxy/0.log"
Jul 01 19:48:10.718178 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:10.718130    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5xzb7_785951ec-d3df-4e9b-89b4-06e66ef5dfe0/init-textfile/0.log"
Jul 01 19:48:13.480034 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.480000    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"]
Jul 01 19:48:13.485480 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.485455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.504999 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.504962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"]
Jul 01 19:48:13.544227 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.544192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-proc\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.544416 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.544253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zpr\" (UniqueName: \"kubernetes.io/projected/673e2474-daa8-4c7c-9599-3daf80df275f-kube-api-access-h6zpr\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.544416 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.544282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-lib-modules\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.544416 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.544395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-podres\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.544577 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.544443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-sys\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645069 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-sys\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-proc\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zpr\" (UniqueName: \"kubernetes.io/projected/673e2474-daa8-4c7c-9599-3daf80df275f-kube-api-access-h6zpr\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-lib-modules\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-sys\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645251 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-proc\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645448 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-podres\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645448 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-lib-modules\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.645448 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.645429    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/673e2474-daa8-4c7c-9599-3daf80df275f-podres\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.654108 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.654072    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zpr\" (UniqueName: \"kubernetes.io/projected/673e2474-daa8-4c7c-9599-3daf80df275f-kube-api-access-h6zpr\") pod \"perf-node-gather-daemonset-4lkhj\" (UID: \"673e2474-daa8-4c7c-9599-3daf80df275f\") " pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.797085 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.796806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:13.957525 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:13.957411    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"]
Jul 01 19:48:14.270727 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:14.270674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj" event={"ID":"673e2474-daa8-4c7c-9599-3daf80df275f","Type":"ContainerStarted","Data":"7e94add792b1ed4c00655491a27c368169860aa02f00adc92b3cf90c730b47ee"}
Jul 01 19:48:14.271149 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:14.270742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj" event={"ID":"673e2474-daa8-4c7c-9599-3daf80df275f","Type":"ContainerStarted","Data":"1a9379b4339ead53af9e74de782995d19a9119595eb817a24e6eacc0e3e48880"}
Jul 01 19:48:14.271149 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:14.270862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:14.289230 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:14.289178    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj" podStartSLOduration=1.289159653 podStartE2EDuration="1.289159653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:48:14.287457302 +0000 UTC m=+1559.388345561" watchObservedRunningTime="2026-07-01 19:48:14.289159653 +0000 UTC m=+1559.390047910"
Jul 01 19:48:15.378590 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:15.378564    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dtzqg_94bd21a2-02a8-4572-bb24-c096e393aa8d/dns/0.log"
Jul 01 19:48:15.408037 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:15.407990    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dtzqg_94bd21a2-02a8-4572-bb24-c096e393aa8d/kube-rbac-proxy/0.log"
Jul 01 19:48:15.597701 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:15.597672    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n42f2_57fde416-4c2f-450b-9be6-46ab9b93e5fa/dns-node-resolver/0.log"
Jul 01 19:48:16.173490 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:16.173463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h65d5_c4b86413-a00d-4b8c-bba9-0b094c5dacdf/node-ca/0.log"
Jul 01 19:48:17.459508 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:17.459484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lvc5r_e3145f21-bf24-4c73-911d-4b0aecb353c4/serve-healthcheck-canary/0.log"
Jul 01 19:48:18.039134 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:18.039106    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrzs5_81a32760-9a0f-4f9b-b430-be68d041bed3/kube-rbac-proxy/0.log"
Jul 01 19:48:18.071705 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:18.071675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrzs5_81a32760-9a0f-4f9b-b430-be68d041bed3/exporter/0.log"
Jul 01 19:48:18.119917 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:18.119890    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lrzs5_81a32760-9a0f-4f9b-b430-be68d041bed3/extractor/0.log"
Jul 01 19:48:20.284418 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.284386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26hz8/perf-node-gather-daemonset-4lkhj"
Jul 01 19:48:20.319650 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.319600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-79d94fb575-qmdwr_76e0cba6-2453-4e6e-9970-d54c216366ab/manager/0.log"
Jul 01 19:48:20.439554 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.439526    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-kn8dc_78e3ce6b-8572-4a0d-b1f4-eebbe425c7ef/manager/0.log"
Jul 01 19:48:20.471511 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.471484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nsdbp_402f27b8-6cf5-4834-a7d3-a214419d46cf/s3-init/0.log"
Jul 01 19:48:20.488723 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.488694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nsdbp_402f27b8-6cf5-4834-a7d3-a214419d46cf/download-hf-model/0.log"
Jul 01 19:48:20.518462 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.518439    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-gwww9_188b027f-7604-4e31-9065-0dffb95bc0a9/s3-tls-init-custom/0.log"
Jul 01 19:48:20.539119 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.539062    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-gwww9_188b027f-7604-4e31-9065-0dffb95bc0a9/download-hf-model/0.log"
Jul 01 19:48:20.568464 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.568441    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-d9vbq_45315d8c-819d-45c3-a88f-d97e5f4b3166/s3-tls-init-serving/0.log"
Jul 01 19:48:20.592597 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.592575    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-d9vbq_45315d8c-819d-45c3-a88f-d97e5f4b3166/download-hf-model/0.log"
Jul 01 19:48:20.699312 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.699279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-l4lkw_e2e06435-0411-4c04-bb3f-41c66994d30e/seaweedfs-tls-custom/0.log"
Jul 01 19:48:20.731692 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:20.731664    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-blkg4_ba0e7eee-9e18-4dac-be6b-9484babfe941/seaweedfs-tls-serving/0.log"
Jul 01 19:48:27.227093 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.227063    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/kube-multus-additional-cni-plugins/0.log"
Jul 01 19:48:27.253841 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.253809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/egress-router-binary-copy/0.log"
Jul 01 19:48:27.277361 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.277335    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/cni-plugins/0.log"
Jul 01 19:48:27.299421 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.299397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/bond-cni-plugin/0.log"
Jul 01 19:48:27.321774 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.321749    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/routeoverride-cni/0.log"
Jul 01 19:48:27.353157 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.353130    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/whereabouts-cni-bincopy/0.log"
Jul 01 19:48:27.375953 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.375925    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgpsw_7b613c19-586f-46d4-a496-bbe115025381/whereabouts-cni/0.log"
Jul 01 19:48:27.410215 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.410184    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-frnld_755375e2-ca71-43d3-8845-2a84b281b425/kube-multus/0.log"
Jul 01 19:48:27.511926 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.511831    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j9pkn_652e707d-3bc9-4c49-a572-bfe0be67203d/network-metrics-daemon/0.log"
Jul 01 19:48:27.534276 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:27.534247    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j9pkn_652e707d-3bc9-4c49-a572-bfe0be67203d/kube-rbac-proxy/0.log"
Jul 01 19:48:28.994055 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:28.994030    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/ovn-controller/0.log"
Jul 01 19:48:29.022754 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.022731    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/ovn-acl-logging/0.log"
Jul 01 19:48:29.071364 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.071334    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/kube-rbac-proxy-node/0.log"
Jul 01 19:48:29.111816 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.111788    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 19:48:29.147019 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.146981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/northd/0.log"
Jul 01 19:48:29.177589 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.177560    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/nbdb/0.log"
Jul 01 19:48:29.211709 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.211671    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/sbdb/0.log"
Jul 01 19:48:29.263290 ip-10-0-131-138 kubenswrapper[2578]: I0701 19:48:29.263201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p568j_af186ab4-8d4a-4bf2-af43-0082dc8280e5/ovnkube-controller/0.log"