Jul 07 17:38:25.903989 ip-10-0-138-144 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 17:38:25.904004 ip-10-0-138-144 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 17:38:25.904013 ip-10-0-138-144 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 17:38:25.904354 ip-10-0-138-144 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 17:38:35.909932 ip-10-0-138-144 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 17:38:35.909952 ip-10-0-138-144 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 48cc2c25dfbb47c391609747ee639b13 --
Jul 07 17:40:38.263434 ip-10-0-138-144 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.739506    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747849    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747867    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747872    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747877    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747881    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747885    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747890    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 17:40:38.856726 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747894    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 17:40:38.832862 ip-10-0-138-144 systemd[1]: Started Kubernetes Kubelet.
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747898    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747901    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747905    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747909    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747913    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747917    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747921    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747924    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747928    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747932    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747935    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747939    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747943    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747946    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747950    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747954    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747957    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747961    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747965    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 17:40:38.858603 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747969    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747974    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747978    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747981    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747985    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.747998    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748002    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748006    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748010    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748014    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748017    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748021    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748024    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748027    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748029    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748033    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748038    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748041    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748045    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748049    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 17:40:38.859800 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748052    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748058    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748065    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748070    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748074    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748078    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748083    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748088    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748092    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748096    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748102    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748118    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748124    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748130    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748135    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748139    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748144    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748148    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748152    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 17:40:38.860707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748156    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748162    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748165    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748169    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748173    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748179    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748184    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748188    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748192    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748196    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748200    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748204    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748208    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748211    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748217    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748221    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748225    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748229    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748232    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 17:40:38.861812 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748235    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748239    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748242    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748247    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748251    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748254    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748258    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748262    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748266    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748270    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748274    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748279    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748283    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.748286    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749288    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749298    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749302    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749306    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749311    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749315    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 17:40:38.862840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749319    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749323    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749328    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749332    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749336    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749340    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749344    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749348    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749352    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749356    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749359    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749364    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749368    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749372    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749376    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749379    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749383    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749387    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749393    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 17:40:38.864228 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749398    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749402    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749407    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749411    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749417    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749420    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749424    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749428    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749432    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749436    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749440    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749444    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749447    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749450    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749452    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749456    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749460    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749463    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749467    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 17:40:38.865256 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749471    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749475    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749479    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749482    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749486    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749491    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749496    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749500    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749504    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749507    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749511    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749515    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749519    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749524    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749528    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749531    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749535    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749538    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749542    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749546    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 17:40:38.866338 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749550    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749554    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749557    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749561    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749565    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749568    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749572    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749576    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749580    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749584    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749587    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749591    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749595    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749601    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749606    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749610    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749614    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749618    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749622    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 17:40:38.867084 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749626    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749630    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749633    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749640    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749644    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749647    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749650    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749653    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749655    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749659    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749666    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749672    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749677    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749682    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749685    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.749689    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750102    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750123    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750128    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 17:40:38.867840 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750131    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750135    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750138    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750141    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750144    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750147    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750150    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750153    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750155    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750158    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750164    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750167    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750169    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750172    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750175    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750178    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750180    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750183    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750187    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 17:40:38.868695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750190    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750193    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750196    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750199    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750202    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750205    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750208    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750210    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750213    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750216    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750219    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750222    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750224    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750226    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750229    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750231    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750234    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750236    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750239    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750241    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 17:40:38.869413 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750244    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750247    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750249    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750253    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750256    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750258    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750260    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750264    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750266    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750269    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750271    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750273    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750276    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750278    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750281    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750283    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750287    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750289    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750292    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750295    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 17:40:38.870123 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750297    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750299    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750302    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750304    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750307    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750310    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750312    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750315    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750317    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750320    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750322    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750324    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750327    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750329    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750331    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750335    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750337    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750341    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750344    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 17:40:38.871037 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750347    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750349    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750352    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750354    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750357    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750359    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750362    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750364    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750367    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750371    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750373    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750376    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750378    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750380    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750383    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750387    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750389    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.750392    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750463    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750469    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 17:40:38.871925 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750475    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750480    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750484    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750487    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750493    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750497    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750500    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750503    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750508    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750511    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750515    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750517    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750520    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750523    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750527    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750530    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750533    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750536    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750539    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750542    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750545    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750550    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750553    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 17:40:38.873272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750556    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750560    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750562    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750565    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750568    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750571    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750574    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750578    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750581    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750584    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750587    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750589    2577 flags.go:64] FLAG: --enable-server="true"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750593    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750597    2577 flags.go:64] FLAG: --event-burst="100"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750600    2577 flags.go:64] FLAG: --event-qps="50"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750603    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750606    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750610    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750614    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750617    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750620    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750623    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750626    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750629    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750633    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 17:40:38.874109 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750636    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750638    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750641    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750644    2577 flags.go:64] FLAG: --feature-gates=""
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750647    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750653    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750655    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750658    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750661    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750664    2577 flags.go:64] FLAG: --help="false"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750667    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-144.ec2.internal"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750671    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750673    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750676    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750679    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750683    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750686    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750688    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750691    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750694    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750697    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750700    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750702    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750705    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 17:40:38.875035 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750710    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750713    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750716    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750719    2577 flags.go:64] FLAG: --lock-file=""
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750721    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750724    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750727    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750732    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750736    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750738    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750741    2577 flags.go:64] FLAG: --logging-format="text"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750743    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750747    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750751    2577 flags.go:64] FLAG: --manifest-url=""
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750754    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750758    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750761    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750765    2577 flags.go:64] FLAG: --max-pods="110"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750768    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750771    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750774    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750776    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750779    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750782    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750785    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 17:40:38.876210 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750793    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750795    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750798    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750801    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750804    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750811    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750813    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750816    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750820    2577 flags.go:64] FLAG: --port="10250"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750823    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750826    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fcd310089e905723"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750829    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750832    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750835    2577 flags.go:64] FLAG: --register-node="true"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750838    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750841    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750845    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750848    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750850    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750854    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750858    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750861    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750864    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750867    2577 flags.go:64] FLAG: --runonce="false"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750870    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 17:40:39.122792 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750873    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750876    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750878    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750881    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750884    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750887    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750890    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750893    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750896    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750899    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750902    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750905    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750907    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750910    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750915    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750919    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750922    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750926    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750929    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750931    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750934    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750937    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750940    2577 flags.go:64] FLAG: --v="2"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750943    2577 flags.go:64] FLAG: --version="false"
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750948    2577 flags.go:64] FLAG: --vmodule=""
Jul 07 17:40:39.142564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750956    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.750959    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751082    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751086    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751091    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751095    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751099    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751103    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751119    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751124    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751127    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751131    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751137    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751142    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751145    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751149    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751153    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751157    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751161    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751164    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 17:40:39.143464 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751168    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751171    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751177    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751181    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751184    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751189    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751192    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751196    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751199    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751203    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751207    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751211    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751217    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751221    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751228    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751232    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751236    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751240    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751244    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751248    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 17:40:39.144129 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751252    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751255    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751259    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751263    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751267    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751271    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751275    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751279    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751283    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751287    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751291    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751295    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751299    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751302    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751307    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751311    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751315    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751318    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751321    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751324    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 17:40:39.144804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751326    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751329    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751332    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751334    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751337    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751340    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751345    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751347    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751350    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751353    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751357    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751360    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751364    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751367    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751370    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751372    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751375    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751377    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751380    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 17:40:39.145458 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751383    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751385    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751388    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751390    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751393    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751396    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751398    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751402    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751404    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751407    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751409    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751412    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751414    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751416    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751419    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751422    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751424    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751427    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751429    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 17:40:39.146093 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751433    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751437    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.751441    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.751991    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.759122    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.759141    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.759162    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.759168    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759249    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759256    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759259    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759263    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759267    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759272    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 17:40:39.146707 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759275    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759278    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759282    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759284    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759287    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759290    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759293    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759295    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759298    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759301    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759304    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759308    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759312    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759316    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759319    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759322    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759324    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759327    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759331    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759334    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 17:40:39.147283 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759337    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759340    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759343    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759346    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759349    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759352    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759355    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759358    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759360    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759363    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759365    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759368    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759371    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759374    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759377    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759380    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759383    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759385    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759388    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759391    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 17:40:39.147897 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759394    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759398    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759402    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759405    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759408    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759411    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759414    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759417    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759420    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759423    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759429    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759432    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759435    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759438    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759441    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759444    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759447    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759450    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759452    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 17:40:39.148540 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759455    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759457    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759460    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759463    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759465    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759468    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759471    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759473    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759476    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759478    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759481    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759484    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759486    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759489    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759492    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759494    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759497    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759500    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759502    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759505    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 17:40:39.149164 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759508    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759511    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759513    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759518    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759521    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759524    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759526    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759529    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759532    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759535    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759538    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759540    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759544    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759546    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.759552    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 17:40:39.149820 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759690    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759696    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759699    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759702    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759705    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759708    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759711    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759714    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759718    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759722    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759724    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759727    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759730    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759733    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759736    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759739    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759741    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759744    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759746    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759749    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 17:40:39.150353 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759753    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759757    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759759    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759763    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759767    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759771    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759774    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759777    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759781    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759785    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759788    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759791    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759794    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759796    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759799    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759802    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759807    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759809    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759812    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759815    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 17:40:39.150974 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759817    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759820    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759823    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759825    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759828    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759831    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759834    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759836    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759839    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759841    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759844    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759847    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759849    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759854    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759856    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759859    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759862    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759865    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759867    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759870    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 17:40:39.151585 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759873    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759876    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759879    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759882    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759884    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759888    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759892    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759895    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759899    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759902    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759905    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759908    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759911    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759913    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759916    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759919    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759921    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759924    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759927    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 17:40:39.152220 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759930    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759932    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759935    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759938    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759940    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759943    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759948    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759951    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759954    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759957    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759960    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759962    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759965    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759968    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759970    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759973    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759976    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759979    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759982    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 17:40:39.152782 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:38.759984    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.759989    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.760117    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.763124    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.764640    2577 server.go:1019] "Starting client certificate rotation"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.764748    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.764782    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.769311    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.773843    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.787309    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.789921    2577 log.go:25] "Validated CRI v1 image API"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.790824    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 17:40:39.373417 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.795428    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 17:40:39.373962 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.797358    2577 fs.go:135] Filesystem UUIDs: map[261118e2-e6bd-4d4a-97c5-9529b5930f98:/dev/nvme0n1p4 369bf838-9992-48b1-8bd3-799c09a7b86c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 07 17:40:39.373962 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.797373    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.803856    2577 manager.go:216] Machine: {Timestamp:2026-07-07 17:40:38.80186731 +0000 UTC m=+0.375814644 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100172 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cee997b8473f3dd63b3829d4d5786 SystemUUID:ec2cee99-7b84-73f3-dd63-b3829d4d5786 BootID:48cc2c25-dfbb-47c3-9160-9747ee639b13 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:72:6c:71:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:72:6c:71:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:0b:44:0b:a0:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.803975    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.804071    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.808530    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.808559    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.809265    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.809276    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.809285    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.811142    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.812529    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815757    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815885    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815917    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815932    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815955    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815970    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.815982    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 17:40:39.374566 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.816362    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.816381    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.820295    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.822766    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.822803    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824122    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824168    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824184    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824193    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824156    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zhfhh"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824204    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824211    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824219    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824225    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.824232    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.825274    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.825713    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.825727    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.826594    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.826617    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.830118    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.830122    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.830137    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.830374    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zhfhh"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.831852    2577 server.go:1268] "Started kubelet"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.831943    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.831977    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.832027    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.833230    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.833420    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.840956    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.844752    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.844757    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.844859    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845339    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845532    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845551    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845582    2577 factory.go:55] Registering systemd factory
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845597    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845676    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845686    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845897    2577 factory.go:153] Registering CRI-O factory
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.845915    2577 factory.go:223] Registration of the crio container factory successfully
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.846008    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.846040    2577 factory.go:103] Registering Raw factory
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.846157    2577 manager.go:1196] Started watching for new ooms in manager
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.846766    2577 manager.go:349] Starting recovery of all containers
Jul 07 17:40:39.375201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.848872    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.860521    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-144.ec2.internal\" not found" node="ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.861817    2577 manager.go:354] Recovery completed
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.864865    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.869353    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.871800    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.871831    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.871847    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.872542    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.872555    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.872575    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.872708    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.874359    2577 policy_none.go:49] "None policy: Start"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.874377    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.874392    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.875389    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.875415    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.875426    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.876009    2577 policy_none.go:47] "Start"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.930864    2577 manager.go:337] "Starting Device Plugin manager"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.931078    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.931092    2577 server.go:87] "Starting device plugin registration server"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.931355    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.931366    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.931486    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.931575    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:38.931584    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.932133    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:38.932178    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.031452    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.032555    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.032585    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.032596    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.032630    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.041828    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.041849    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-144.ec2.internal\": node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.058876    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.090323    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.090368    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.090396    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.090436    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.092913    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.160053    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.192661    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"]
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.192760    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.194515    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.376713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.194544    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.194556    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196085    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196252    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196852    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196859    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196877    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196888    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196876    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.196971    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.198426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.198450    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.199104    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.199126    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.199139    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeHasSufficientPID"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.217816    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-144.ec2.internal\" not found" node="ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.222375    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-144.ec2.internal\" not found" node="ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.247182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33f28cff3cad66dfe5aa1e747b9f44c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-144.ec2.internal\" (UID: \"d33f28cff3cad66dfe5aa1e747b9f44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.247205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f89bf03cbf59db5897f34916fe53bd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"4f89bf03cbf59db5897f34916fe53bd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.247224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f89bf03cbf59db5897f34916fe53bd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"4f89bf03cbf59db5897f34916fe53bd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.261252    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.347862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33f28cff3cad66dfe5aa1e747b9f44c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-144.ec2.internal\" (UID: \"d33f28cff3cad66dfe5aa1e747b9f44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.347896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f89bf03cbf59db5897f34916fe53bd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"4f89bf03cbf59db5897f34916fe53bd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.347918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f89bf03cbf59db5897f34916fe53bd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"4f89bf03cbf59db5897f34916fe53bd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.347972    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d33f28cff3cad66dfe5aa1e747b9f44c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-144.ec2.internal\" (UID: \"d33f28cff3cad66dfe5aa1e747b9f44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.347988    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f89bf03cbf59db5897f34916fe53bd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"4f89bf03cbf59db5897f34916fe53bd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378899 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.348022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f89bf03cbf59db5897f34916fe53bd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal\" (UID: \"4f89bf03cbf59db5897f34916fe53bd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.378899 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.361964    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.559542 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.462985    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.559542 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.524863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.559542 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.526056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:39.806534 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.725267    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.806534 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.763494    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 17:40:39.806534 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.763682    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 17:40:39.806534 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.763688    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 17:40:39.826142 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.825321    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:39.932039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.835338    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 17:35:39 +0000 UTC" deadline="2028-03-23 02:05:45.652865732 +0000 UTC"
Jul 07 17:40:39.932039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:39.835363    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14984h25m5.817505795s"
Jul 07 17:40:39.932039 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:39.926323    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-144.ec2.internal\" not found"
Jul 07 17:40:40.061942 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.004375    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 17:40:40.061942 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.045573    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:40.061942 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.056175    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 17:40:40.061942 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.058339    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal"
Jul 07 17:40:40.215913 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.071622    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 17:40:40.329102 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.324099    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 17:40:40.784216 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:40.784161    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33f28cff3cad66dfe5aa1e747b9f44c.slice/crio-3c7d2a5866a4ada847f21f68b36d1a4ca2011c5446309b1cc34d181abcf7b619 WatchSource:0}: Error finding container 3c7d2a5866a4ada847f21f68b36d1a4ca2011c5446309b1cc34d181abcf7b619: Status 404 returned error can't find the container with id 3c7d2a5866a4ada847f21f68b36d1a4ca2011c5446309b1cc34d181abcf7b619
Jul 07 17:40:40.784561 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:40.784542    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f89bf03cbf59db5897f34916fe53bd2.slice/crio-01012d1e6c08667758d13102dfb4e7b7b0996c0ce98c3daa4c3a58e55e4fa779 WatchSource:0}: Error finding container 01012d1e6c08667758d13102dfb4e7b7b0996c0ce98c3daa4c3a58e55e4fa779: Status 404 returned error can't find the container with id 01012d1e6c08667758d13102dfb4e7b7b0996c0ce98c3daa4c3a58e55e4fa779
Jul 07 17:40:40.788600 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.788583    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:40:40.818081 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.818052    2577 apiserver.go:52] "Watching apiserver"
Jul 07 17:40:40.840975 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.840954    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 17:40:40.841332 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.841312    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl","openshift-cluster-node-tuning-operator/tuned-4wdxl","openshift-image-registry/node-ca-8kntg","openshift-multus/multus-additional-cni-plugins-pd4lj","openshift-multus/multus-dzkll","openshift-network-diagnostics/network-check-target-tnmvh","kube-system/konnectivity-agent-glpm8","openshift-dns/node-resolver-fv45b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal","openshift-multus/network-metrics-daemon-6s6cr","openshift-network-operator/iptables-alerter-w27zq","openshift-ovn-kubernetes/ovnkube-node-bw4c2","kube-system/global-pull-secret-syncer-4rd2v"]
Jul 07 17:40:40.845888 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.844017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:40.845888 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:40.844122    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:40.846057 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.845878    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 17:40:40.846544 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.846520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.847144 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.847128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.848610 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.848590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:40.849469 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.849453    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.849542 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.849458    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 17:40:40.849742 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.849720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:40.850204 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.850185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2gpfj\""
Jul 07 17:40:40.850309 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.850290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2gdjk\""
Jul 07 17:40:40.850366 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.850352    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 17:40:40.850855 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.850836    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 17:40:40.851051 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.851034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzkll"
Jul 07 17:40:40.852198 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.852182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:40.853861 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.853611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:40.853861 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:40.853706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:40.853951 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.853868    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.854194 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854175    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.854303 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.854449 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.854665 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 17:40:40.854756 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.854756 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854723    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 17:40:40.854860 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 17:40:40.854915 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.854915 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854897    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 17:40:40.855015 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.854930    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.855069 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855015    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 17:40:40.855147 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9976q\""
Jul 07 17:40:40.855229 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:40.855349 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855319    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.855436 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qc82t\""
Jul 07 17:40:40.855708 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855691    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.855820 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855721    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 17:40:40.855820 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4vllp\""
Jul 07 17:40:40.855930 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.855830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pg8gn\""
Jul 07 17:40:40.856389 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.856376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:40.857781 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.857765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:40.857860 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:40.857842    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:40.859054 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.859039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:40.861554 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.861527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 17:40:40.861554 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.861532    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.862359 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.862308    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.862359 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.862321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 17:40:40.862518 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.862372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rz448\""
Jul 07 17:40:40.862518 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.862396    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 17:40:40.862518 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.862428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hpwnw\""
Jul 07 17:40:40.863227 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.863213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 17:40:40.863409 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.863396    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lgncq\""
Jul 07 17:40:40.863476 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.863451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 17:40:40.863774 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.863757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 17:40:40.866803 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.866786    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 17:40:40.931882 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.931842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysctl-conf\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.931890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-var-lib-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.931931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.931966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-cni-netd\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.931986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c610d36a-d68f-4a47-90a0-62f695ccace9-ovn-node-metrics-cert\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysconfig\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-systemd-units\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-ovnkube-config\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-modprobe-d\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-slash\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-node-log\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-env-overrides\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-sys\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-etc-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-kubernetes\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-run\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-var-lib-kubelet\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-ovn\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-log-socket\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-run-ovn-kubernetes\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.932567 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-ovnkube-script-lib\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftn7h\" (UniqueName: \"kubernetes.io/projected/c610d36a-d68f-4a47-90a0-62f695ccace9-kube-api-access-ftn7h\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-lib-modules\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-tmp\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-kubelet\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-systemd\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-cni-bin\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysctl-d\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-systemd\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97zq\" (UniqueName: \"kubernetes.io/projected/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-kube-api-access-p97zq\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-run-netns\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-host\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.933026 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.932692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-tuned\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:40.946160 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:40.946142    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 17:40:41.012396 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.012368    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qgkkt"
Jul 07 17:40:41.033486 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-socket-dir-parent\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033486 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-cni-multus\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033486 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-kubelet\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033617 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmkn\" (UniqueName: \"kubernetes.io/projected/e6523d94-f705-49fb-ac51-83169974230c-kube-api-access-sbmkn\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:41.033617 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-run\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.033617 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-var-lib-kubelet\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.033617 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-cni-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e039226-8e03-42d0-b084-f552e7b4c2b5-tmp-dir\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-ovnkube-script-lib\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-run\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-system-cni-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-var-lib-kubelet\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-hostroot\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-cni-bin\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1d094db-aab0-4daa-8ac6-39e13bd8d3ee-konnectivity-ca\") pod \"konnectivity-agent-glpm8\" (UID: \"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee\") " pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctvc\" (UniqueName: \"kubernetes.io/projected/d2e560e2-d355-45c9-90ab-7c6baca64f5b-kube-api-access-9ctvc\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-device-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbnm\" (UniqueName: \"kubernetes.io/projected/626a6665-283d-4b4e-94ab-d83f79c5d17c-kube-api-access-tmbnm\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.033974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-kubelet\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-cni-bin\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-systemd\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.033999    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-kubelet\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-cnibin\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-daemon-config\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034034    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-cni-bin\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2e560e2-d355-45c9-90ab-7c6baca64f5b-host\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034055    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-systemd\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-host\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-tuned\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-host\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-dbus\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e039226-8e03-42d0-b084-f552e7b4c2b5-hosts-file\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.034216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-socket-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-sys-fs\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/626a6665-283d-4b4e-94ab-d83f79c5d17c-iptables-alerter-script\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysctl-conf\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034401    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034418    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034384    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-ovnkube-script-lib\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034435    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysctl-conf\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2e560e2-d355-45c9-90ab-7c6baca64f5b-serviceca\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-systemd-units\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-ovnkube-config\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-systemd-units\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626a6665-283d-4b4e-94ab-d83f79c5d17c-host-slash\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-os-release\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.034703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grm7h\" (UniqueName: \"kubernetes.io/projected/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-kube-api-access-grm7h\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-node-log\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-sys\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-multus-certs\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-registration-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-sys\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-etc-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-node-log\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034780    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-etc-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-kubernetes\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-cni-binary-copy\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-k8s-cni-cncf-io\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034859    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-kubernetes\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1d094db-aab0-4daa-8ac6-39e13bd8d3ee-agent-certs\") pod \"konnectivity-agent-glpm8\" (UID: \"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee\") " pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-ovn\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-log-socket\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034972    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-ovnkube-config\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.035378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.034986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-run-ovn-kubernetes\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035002    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-ovn\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035019    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-run-ovn-kubernetes\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftn7h\" (UniqueName: \"kubernetes.io/projected/c610d36a-d68f-4a47-90a0-62f695ccace9-kube-api-access-ftn7h\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-log-socket\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-lib-modules\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-tmp\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-etc-kubernetes\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-systemd\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysctl-d\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035172    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-lib-modules\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97zq\" (UniqueName: \"kubernetes.io/projected/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-kube-api-access-p97zq\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-system-cni-dir\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035221    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-run-systemd\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-run-netns\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-kubelet-config\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.036139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-cni-binary-copy\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035360    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysctl-d\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcksq\" (UniqueName: \"kubernetes.io/projected/988895c8-9fa3-482b-97a1-9815c60b8afa-kube-api-access-wcksq\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-run-netns\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-var-lib-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-cni-netd\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c610d36a-d68f-4a47-90a0-62f695ccace9-ovn-node-metrics-cert\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035543    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-cni-netd\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-var-lib-openvswitch\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysconfig\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-sysconfig\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v578x\" (UniqueName: \"kubernetes.io/projected/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-kube-api-access-v578x\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.036720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-netns\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-os-release\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6d66\" (UniqueName: \"kubernetes.io/projected/0e039226-8e03-42d0-b084-f552e7b4c2b5-kube-api-access-w6d66\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035803    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qgkkt"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-modprobe-d\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-conf-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035903    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-modprobe-d\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-cnibin\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-slash\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.035956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-env-overrides\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.036193    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c610d36a-d68f-4a47-90a0-62f695ccace9-host-slash\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.037214 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.036331    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c610d36a-d68f-4a47-90a0-62f695ccace9-env-overrides\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.037908 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.037886    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-etc-tuned\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.037982 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.037922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-tmp\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.038039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.038026    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c610d36a-d68f-4a47-90a0-62f695ccace9-ovn-node-metrics-cert\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.059619 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.059601    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:40:41.059619 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.059623    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:40:41.059772 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.059634    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:41.059772 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.059685    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:41.559670344 +0000 UTC m=+3.133617663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:41.062755 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.062737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97zq\" (UniqueName: \"kubernetes.io/projected/c8101c26-b5b7-4d36-85f2-ed6416f72bf4-kube-api-access-p97zq\") pod \"tuned-4wdxl\" (UID: \"c8101c26-b5b7-4d36-85f2-ed6416f72bf4\") " pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.063764 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.063748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftn7h\" (UniqueName: \"kubernetes.io/projected/c610d36a-d68f-4a47-90a0-62f695ccace9-kube-api-access-ftn7h\") pod \"ovnkube-node-bw4c2\" (UID: \"c610d36a-d68f-4a47-90a0-62f695ccace9\") " pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.082272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.082251    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 17:40:41.095994 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.095932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" event={"ID":"4f89bf03cbf59db5897f34916fe53bd2","Type":"ContainerStarted","Data":"01012d1e6c08667758d13102dfb4e7b7b0996c0ce98c3daa4c3a58e55e4fa779"}
Jul 07 17:40:41.097002 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.096975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal" event={"ID":"d33f28cff3cad66dfe5aa1e747b9f44c","Type":"ContainerStarted","Data":"3c7d2a5866a4ada847f21f68b36d1a4ca2011c5446309b1cc34d181abcf7b619"}
Jul 07 17:40:41.136635 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-cni-binary-copy\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.136635 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136612    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 17:40:41.136860 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.136860 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcksq\" (UniqueName: \"kubernetes.io/projected/988895c8-9fa3-482b-97a1-9815c60b8afa-kube-api-access-wcksq\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.136972 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.136972 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.136972 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v578x\" (UniqueName: \"kubernetes.io/projected/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-kube-api-access-v578x\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.136972 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-netns\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.136972 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-os-release\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.136992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6d66\" (UniqueName: \"kubernetes.io/projected/0e039226-8e03-42d0-b084-f552e7b4c2b5-kube-api-access-w6d66\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-conf-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137034    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-netns\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-cnibin\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.137073    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137100    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-conf-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137117    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-cnibin\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-socket-dir-parent\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.137189    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:41.637169526 +0000 UTC m=+3.211116867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-os-release\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137211    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-socket-dir-parent\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-cni-multus\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137266 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-kubelet\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmkn\" (UniqueName: \"kubernetes.io/projected/e6523d94-f705-49fb-ac51-83169974230c-kube-api-access-sbmkn\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-cni-binary-copy\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137338    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-kubelet\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137413    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-cni-multus\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-cni-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e039226-8e03-42d0-b084-f552e7b4c2b5-tmp-dir\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-system-cni-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137571    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-hostroot\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-hostroot\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-cni-bin\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137639    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-var-lib-cni-bin\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.137927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1d094db-aab0-4daa-8ac6-39e13bd8d3ee-konnectivity-ca\") pod \"konnectivity-agent-glpm8\" (UID: \"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee\") " pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137650    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-cni-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-system-cni-dir\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctvc\" (UniqueName: \"kubernetes.io/projected/d2e560e2-d355-45c9-90ab-7c6baca64f5b-kube-api-access-9ctvc\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-device-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbnm\" (UniqueName: \"kubernetes.io/projected/626a6665-283d-4b4e-94ab-d83f79c5d17c-kube-api-access-tmbnm\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-cnibin\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-device-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-daemon-config\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e039226-8e03-42d0-b084-f552e7b4c2b5-tmp-dir\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2e560e2-d355-45c9-90ab-7c6baca64f5b-host\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-cnibin\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137849    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2e560e2-d355-45c9-90ab-7c6baca64f5b-host\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-dbus\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e039226-8e03-42d0-b084-f552e7b4c2b5-hosts-file\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137894    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-dbus\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-socket-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-sys-fs\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.138700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/626a6665-283d-4b4e-94ab-d83f79c5d17c-iptables-alerter-script\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137953    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e039226-8e03-42d0-b084-f552e7b4c2b5-hosts-file\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.137997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2e560e2-d355-45c9-90ab-7c6baca64f5b-serviceca\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-sys-fs\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138035    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/988895c8-9fa3-482b-97a1-9815c60b8afa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626a6665-283d-4b4e-94ab-d83f79c5d17c-host-slash\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.138145    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-socket-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-os-release\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138169    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d1d094db-aab0-4daa-8ac6-39e13bd8d3ee-konnectivity-ca\") pod \"konnectivity-agent-glpm8\" (UID: \"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee\") " pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.138211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:41.638193433 +0000 UTC m=+3.212140770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138214    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-os-release\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grm7h\" (UniqueName: \"kubernetes.io/projected/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-kube-api-access-grm7h\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138249    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626a6665-283d-4b4e-94ab-d83f79c5d17c-host-slash\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-multus-certs\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139239 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-registration-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138297    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-multus-certs\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138265    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-cni-binary-copy\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138337    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-registration-dir\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-k8s-cni-cncf-io\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1d094db-aab0-4daa-8ac6-39e13bd8d3ee-agent-certs\") pod \"konnectivity-agent-glpm8\" (UID: \"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee\") " pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138408    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-host-run-k8s-cni-cncf-io\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-etc-kubernetes\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-system-cni-dir\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138459    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-multus-daemon-config\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-etc-kubernetes\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138477    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2e560e2-d355-45c9-90ab-7c6baca64f5b-serviceca\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-kubelet-config\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/626a6665-283d-4b4e-94ab-d83f79c5d17c-iptables-alerter-script\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138576    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-kubelet-config\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138609    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/988895c8-9fa3-482b-97a1-9815c60b8afa-system-cni-dir\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.139766 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.138776    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-cni-binary-copy\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.140659 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.140644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d1d094db-aab0-4daa-8ac6-39e13bd8d3ee-agent-certs\") pod \"konnectivity-agent-glpm8\" (UID: \"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee\") " pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.157750 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.157723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:40:41.159386 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.159366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v578x\" (UniqueName: \"kubernetes.io/projected/f453a5d1-9bee-483a-9191-c73cc2e4eaaa-kube-api-access-v578x\") pod \"aws-ebs-csi-driver-node-x2dtl\" (UID: \"f453a5d1-9bee-483a-9191-c73cc2e4eaaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.160318 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.160303    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcksq\" (UniqueName: \"kubernetes.io/projected/988895c8-9fa3-482b-97a1-9815c60b8afa-kube-api-access-wcksq\") pod \"multus-additional-cni-plugins-pd4lj\" (UID: \"988895c8-9fa3-482b-97a1-9815c60b8afa\") " pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.163425 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.163407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmkn\" (UniqueName: \"kubernetes.io/projected/e6523d94-f705-49fb-ac51-83169974230c-kube-api-access-sbmkn\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:41.163979 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.163961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wdxl"
Jul 07 17:40:41.166361 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.166343    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6d66\" (UniqueName: \"kubernetes.io/projected/0e039226-8e03-42d0-b084-f552e7b4c2b5-kube-api-access-w6d66\") pod \"node-resolver-fv45b\" (UID: \"0e039226-8e03-42d0-b084-f552e7b4c2b5\") " pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.166492 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.166469    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbnm\" (UniqueName: \"kubernetes.io/projected/626a6665-283d-4b4e-94ab-d83f79c5d17c-kube-api-access-tmbnm\") pod \"iptables-alerter-w27zq\" (UID: \"626a6665-283d-4b4e-94ab-d83f79c5d17c\") " pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.167363 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.167339    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc610d36a_d68f_4a47_90a0_62f695ccace9.slice/crio-fe61a08a0c20f4bd78591f8a9f420a7a86f19596653020f504d281af95b7be58 WatchSource:0}: Error finding container fe61a08a0c20f4bd78591f8a9f420a7a86f19596653020f504d281af95b7be58: Status 404 returned error can't find the container with id fe61a08a0c20f4bd78591f8a9f420a7a86f19596653020f504d281af95b7be58
Jul 07 17:40:41.167443 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.167390    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grm7h\" (UniqueName: \"kubernetes.io/projected/0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf-kube-api-access-grm7h\") pod \"multus-dzkll\" (UID: \"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf\") " pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.168226 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.168211    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctvc\" (UniqueName: \"kubernetes.io/projected/d2e560e2-d355-45c9-90ab-7c6baca64f5b-kube-api-access-9ctvc\") pod \"node-ca-8kntg\" (UID: \"d2e560e2-d355-45c9-90ab-7c6baca64f5b\") " pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.170704 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.170687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kntg"
Jul 07 17:40:41.172124 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.172108    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8101c26_b5b7_4d36_85f2_ed6416f72bf4.slice/crio-14a1273435e1ffd78fd6ce639cce4110dcb8b3c0704c048e783b91ce374a7699 WatchSource:0}: Error finding container 14a1273435e1ffd78fd6ce639cce4110dcb8b3c0704c048e783b91ce374a7699: Status 404 returned error can't find the container with id 14a1273435e1ffd78fd6ce639cce4110dcb8b3c0704c048e783b91ce374a7699
Jul 07 17:40:41.175344 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.175321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pd4lj"
Jul 07 17:40:41.179942 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.179920    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e560e2_d355_45c9_90ab_7c6baca64f5b.slice/crio-6a5480e3c9436729cde82e3a6f5a8187b373981193e0ae60b60c051faefb2cfc WatchSource:0}: Error finding container 6a5480e3c9436729cde82e3a6f5a8187b373981193e0ae60b60c051faefb2cfc: Status 404 returned error can't find the container with id 6a5480e3c9436729cde82e3a6f5a8187b373981193e0ae60b60c051faefb2cfc
Jul 07 17:40:41.182630 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.182611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzkll"
Jul 07 17:40:41.182930 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.182867    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988895c8_9fa3_482b_97a1_9815c60b8afa.slice/crio-d69909c0091aff5ed8ea40bf3076850669091f44afcfdf66f89413d415068eb4 WatchSource:0}: Error finding container d69909c0091aff5ed8ea40bf3076850669091f44afcfdf66f89413d415068eb4: Status 404 returned error can't find the container with id d69909c0091aff5ed8ea40bf3076850669091f44afcfdf66f89413d415068eb4
Jul 07 17:40:41.188073 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.188056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fv45b"
Jul 07 17:40:41.193195 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.193174    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad89e8b_8da5_44a0_aedc_fe8a67ca0ccf.slice/crio-bb49e50e2ca0232c50fdaafc35bb42480a09adb0679e292770d05195422cc401 WatchSource:0}: Error finding container bb49e50e2ca0232c50fdaafc35bb42480a09adb0679e292770d05195422cc401: Status 404 returned error can't find the container with id bb49e50e2ca0232c50fdaafc35bb42480a09adb0679e292770d05195422cc401
Jul 07 17:40:41.194487 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.194471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl"
Jul 07 17:40:41.196195 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.196172    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e039226_8e03_42d0_b084_f552e7b4c2b5.slice/crio-70577ad2540c616ec5e8f6f708dd782c637d8f337576abcc22b03c884d2ede6e WatchSource:0}: Error finding container 70577ad2540c616ec5e8f6f708dd782c637d8f337576abcc22b03c884d2ede6e: Status 404 returned error can't find the container with id 70577ad2540c616ec5e8f6f708dd782c637d8f337576abcc22b03c884d2ede6e
Jul 07 17:40:41.199635 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.199550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:40:41.202271 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.202250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf453a5d1_9bee_483a_9191_c73cc2e4eaaa.slice/crio-b83e91b1cbad875e5d41e14aaec4a8d4c2dd916484ee14d235526a9b64d12e47 WatchSource:0}: Error finding container b83e91b1cbad875e5d41e14aaec4a8d4c2dd916484ee14d235526a9b64d12e47: Status 404 returned error can't find the container with id b83e91b1cbad875e5d41e14aaec4a8d4c2dd916484ee14d235526a9b64d12e47
Jul 07 17:40:41.205625 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.205609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w27zq"
Jul 07 17:40:41.216485 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.216469    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d094db_aab0_4daa_8ac6_39e13bd8d3ee.slice/crio-7dce7ae564125c1357659556d231c0bcdcdc99ba10e234d18fda1f4776a946a9 WatchSource:0}: Error finding container 7dce7ae564125c1357659556d231c0bcdcdc99ba10e234d18fda1f4776a946a9: Status 404 returned error can't find the container with id 7dce7ae564125c1357659556d231c0bcdcdc99ba10e234d18fda1f4776a946a9
Jul 07 17:40:41.219206 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:40:41.219176    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626a6665_283d_4b4e_94ab_d83f79c5d17c.slice/crio-6eacb03b7e8e3c498c9161ea5c9ca2aadfe3520cd0b4659b56553568f80d4fbd WatchSource:0}: Error finding container 6eacb03b7e8e3c498c9161ea5c9ca2aadfe3520cd0b4659b56553568f80d4fbd: Status 404 returned error can't find the container with id 6eacb03b7e8e3c498c9161ea5c9ca2aadfe3520cd0b4659b56553568f80d4fbd
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.643530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.643581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:41.643629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643694    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643732    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643771    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:40:41.643808 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643791    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:42.643771769 +0000 UTC m=+4.217719104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:41.644346 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643834    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:42.643822775 +0000 UTC m=+4.217770103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:41.644346 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643793    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:40:41.644346 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643895    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:41.644346 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:41.643948    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:42.643930137 +0000 UTC m=+4.217877475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:42.039313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.039267    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:35:41 +0000 UTC" deadline="2028-02-05 19:47:15.073386944 +0000 UTC"
Jul 07 17:40:42.039313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.039310    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13874h6m33.034081176s"
Jul 07 17:40:42.090775 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.090749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:42.090943 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.090879    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:42.091296 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.091275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:42.091393 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.091370    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:42.123964 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.123683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerStarted","Data":"d69909c0091aff5ed8ea40bf3076850669091f44afcfdf66f89413d415068eb4"}
Jul 07 17:40:42.137507 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.137472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kntg" event={"ID":"d2e560e2-d355-45c9-90ab-7c6baca64f5b","Type":"ContainerStarted","Data":"6a5480e3c9436729cde82e3a6f5a8187b373981193e0ae60b60c051faefb2cfc"}
Jul 07 17:40:42.155671 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.155583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wdxl" event={"ID":"c8101c26-b5b7-4d36-85f2-ed6416f72bf4","Type":"ContainerStarted","Data":"14a1273435e1ffd78fd6ce639cce4110dcb8b3c0704c048e783b91ce374a7699"}
Jul 07 17:40:42.179559 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.179521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w27zq" event={"ID":"626a6665-283d-4b4e-94ab-d83f79c5d17c","Type":"ContainerStarted","Data":"6eacb03b7e8e3c498c9161ea5c9ca2aadfe3520cd0b4659b56553568f80d4fbd"}
Jul 07 17:40:42.194536 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.194497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl" event={"ID":"f453a5d1-9bee-483a-9191-c73cc2e4eaaa","Type":"ContainerStarted","Data":"b83e91b1cbad875e5d41e14aaec4a8d4c2dd916484ee14d235526a9b64d12e47"}
Jul 07 17:40:42.203946 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.203909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fv45b" event={"ID":"0e039226-8e03-42d0-b084-f552e7b4c2b5","Type":"ContainerStarted","Data":"70577ad2540c616ec5e8f6f708dd782c637d8f337576abcc22b03c884d2ede6e"}
Jul 07 17:40:42.219840 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.219745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"fe61a08a0c20f4bd78591f8a9f420a7a86f19596653020f504d281af95b7be58"}
Jul 07 17:40:42.228858 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.228748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-glpm8" event={"ID":"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee","Type":"ContainerStarted","Data":"7dce7ae564125c1357659556d231c0bcdcdc99ba10e234d18fda1f4776a946a9"}
Jul 07 17:40:42.235349 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.235319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzkll" event={"ID":"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf","Type":"ContainerStarted","Data":"bb49e50e2ca0232c50fdaafc35bb42480a09adb0679e292770d05195422cc401"}
Jul 07 17:40:42.653040 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.653001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:42.653244 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.653059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:42.653244 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:42.653122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:42.653353 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653252    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:42.653353 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653317    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:44.653298157 +0000 UTC m=+6.227245491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:42.653752 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653730    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:42.653852 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:44.65376965 +0000 UTC m=+6.227716982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:42.653919 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653869    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:40:42.653919 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653886    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:40:42.653919 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653899    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:42.654062 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:42.653934    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:44.653922463 +0000 UTC m=+6.227869788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:43.040329 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:43.040264    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:35:41 +0000 UTC" deadline="2028-03-18 06:25:23.61995298 +0000 UTC"
Jul 07 17:40:43.040329 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:43.040327    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14868h44m40.57963024s"
Jul 07 17:40:43.094052 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:43.093453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:43.094052 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:43.093576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:44.091205 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:44.091165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:44.091666 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.091318    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:44.091666 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:44.091165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:44.091808 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.091768    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:44.670790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:44.670858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:44.670891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.670963    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671014    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671025    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671037    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:48.671022755 +0000 UTC m=+10.244970079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671041    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671055    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671074    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:48.671057642 +0000 UTC m=+10.245004969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:44.671130 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:44.671095    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:48.671082749 +0000 UTC m=+10.245030073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:45.091132 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:45.091080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:45.091323 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:45.091221    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:46.090849 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:46.090816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:46.091029 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:46.090831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:46.091029 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:46.090976    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:46.091180 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:46.091035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:47.090894 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:47.090683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:47.090894 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:47.090813    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:48.091673 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:48.091638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:48.092129 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.091765    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:48.092200 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:48.092186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:48.092328 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.092296    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:48.700986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:48.701059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:48.701092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701245    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701266    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701280    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701297    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701343    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:56.701321486 +0000 UTC m=+18.275268808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701298    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701363    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:40:56.701353427 +0000 UTC m=+18.275300748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:48.701420 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:48.701388    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:40:56.701371515 +0000 UTC m=+18.275318838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:49.092505 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:49.092457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:49.092964 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:49.092581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:50.091150 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:50.091099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:50.091330 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:50.091099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:50.091330 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:50.091231    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:50.091330 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:50.091292    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:51.091390 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:51.091356    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:51.091850 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:51.091486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:52.090625 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:52.090594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:52.090786 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:52.090594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:52.090786 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:52.090703    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:52.090786 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:52.090777    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:53.091254 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:53.091221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:53.091732 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:53.091355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:54.090880 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:54.090842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:54.090880 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:54.090881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:54.091137 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:54.090991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:54.091137 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:54.091080    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:55.091124 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:55.091067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:55.091517 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:55.091238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:56.090915 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:56.090882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:56.091097 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:56.090921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:56.091097 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.090997    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:56.091383 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.091101    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:56.758176 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:56.758137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:56.758222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:56.758245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758289    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758342    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758355    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758363    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:41:12.758343077 +0000 UTC m=+34.332290399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758368    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:40:56.758371 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758378    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:56.758751 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758400    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:12.758382486 +0000 UTC m=+34.332329819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:40:56.758751 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:56.758420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:41:12.758409698 +0000 UTC m=+34.332357024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:40:57.091555 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:57.091473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:57.091992 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:57.091611    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:40:58.091508 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:58.091475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:40:58.091701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:58.091519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:40:58.091701 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:58.091611    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:40:58.092093 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:58.091713    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:40:59.091762 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:40:59.091731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:40:59.092179 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:40:59.091841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:00.091485 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:00.091448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:00.091666 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:00.091458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:00.091666 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:00.091569    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:00.091769 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:00.091688    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:01.091751 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.091589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:01.092122 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:01.091861    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:01.283919 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.283843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal" event={"ID":"d33f28cff3cad66dfe5aa1e747b9f44c","Type":"ContainerStarted","Data":"be0b19b17b73a67666a21c55d2c06dcbf414d05ec6364bcdbc532fe921d606d5"}
Jul 07 17:41:01.286629 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.286536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzkll" event={"ID":"0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf","Type":"ContainerStarted","Data":"4c8c3118ff9a9abf4e0c76e7ab4cb3cf0be63daf0c9a5dc1f54129fdc5fc0007"}
Jul 07 17:41:01.288878 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.288756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wdxl" event={"ID":"c8101c26-b5b7-4d36-85f2-ed6416f72bf4","Type":"ContainerStarted","Data":"62b0febe2dcf076f9d353b9a716a02724d0777a9be7eed2025ffd700c19e58e9"}
Jul 07 17:41:01.308577 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.308407    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-144.ec2.internal" podStartSLOduration=21.308390026 podStartE2EDuration="21.308390026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:01.307741059 +0000 UTC m=+22.881688402" watchObservedRunningTime="2026-07-07 17:41:01.308390026 +0000 UTC m=+22.882337366"
Jul 07 17:41:01.358699 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.358640    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dzkll" podStartSLOduration=2.652493315 podStartE2EDuration="22.358618495s" totalImagesPullingTime="19.70612518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:01.336298139 +0000 UTC m=+22.910245481" watchObservedRunningTime="2026-07-07 17:41:01.358618495 +0000 UTC m=+22.932565837"
Jul 07 17:41:01.359544 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:01.359503    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wdxl" podStartSLOduration=2.647462275 podStartE2EDuration="22.359487422s" totalImagesPullingTime="19.712025147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:01.358561476 +0000 UTC m=+22.932508817" watchObservedRunningTime="2026-07-07 17:41:01.359487422 +0000 UTC m=+22.933434766"
Jul 07 17:41:02.090995 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.090757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:02.091154 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.090798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:02.091154 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:02.091071    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:02.091270 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:02.091212    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:02.291757 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.291725    2577 generic.go:356] "Generic (PLEG): container finished" podID="4f89bf03cbf59db5897f34916fe53bd2" containerID="f006835e52273f11b2d337b4144b9cb382e7bfa2ae13afc947b730e76c108386" exitCode=0
Jul 07 17:41:02.292439 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.291804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" event={"ID":"4f89bf03cbf59db5897f34916fe53bd2","Type":"ContainerDied","Data":"f006835e52273f11b2d337b4144b9cb382e7bfa2ae13afc947b730e76c108386"}
Jul 07 17:41:02.293619 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.293409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl" event={"ID":"f453a5d1-9bee-483a-9191-c73cc2e4eaaa","Type":"ContainerStarted","Data":"f549dc6ff34463bf9e4bae6e013a4fc045fe692b33a833a833e64bc415c53567"}
Jul 07 17:41:02.294685 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.294663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fv45b" event={"ID":"0e039226-8e03-42d0-b084-f552e7b4c2b5","Type":"ContainerStarted","Data":"2abc17ac3e213a543e462e40303d84fbbc148090f0cda531f340dcc79180bf5c"}
Jul 07 17:41:02.297016 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.296998    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:41:02.297341 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297322    2577 generic.go:356] "Generic (PLEG): container finished" podID="c610d36a-d68f-4a47-90a0-62f695ccace9" containerID="82da5823a19964d699b770f694e75ba6231060649ff338b8981ae571258ef22c" exitCode=1
Jul 07 17:41:02.297411 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"83b4fe38a9133e524ea46e7dfaf51f61d68be3f6220ef61509176fd6ce7b347f"}
Jul 07 17:41:02.297468 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"1eedb1ec7e44c4b3158709e931ff820b00e082ffc1976976f6a614d3df5ac766"}
Jul 07 17:41:02.297520 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"28606ee1a7381d98644bb7eef2fc0049c249be98774ef7ce40b20f0f9d053c74"}
Jul 07 17:41:02.297520 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"3d8f5ea2a31784a4e7f85e99f4a4bf4a0936532d9be98390645dd94941c9bdc8"}
Jul 07 17:41:02.297600 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerDied","Data":"82da5823a19964d699b770f694e75ba6231060649ff338b8981ae571258ef22c"}
Jul 07 17:41:02.297600 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.297575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"71312c1e0dcb5fe67fab94a631bea2ea0c056901b677ab53486dfcf1e1d7b8b5"}
Jul 07 17:41:02.298690 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.298668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-glpm8" event={"ID":"d1d094db-aab0-4daa-8ac6-39e13bd8d3ee","Type":"ContainerStarted","Data":"1b3d8c4b5613c8fb36d9bfe9c9c0e941a655da6020461b7d9ebceff7700ece4f"}
Jul 07 17:41:02.299871 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.299852    2577 generic.go:356] "Generic (PLEG): container finished" podID="988895c8-9fa3-482b-97a1-9815c60b8afa" containerID="1ecddc584544474568f0f7bf5563681c69c70cb87a26b682ef478fea92aac5a8" exitCode=0
Jul 07 17:41:02.299945 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.299928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerDied","Data":"1ecddc584544474568f0f7bf5563681c69c70cb87a26b682ef478fea92aac5a8"}
Jul 07 17:41:02.301226 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.301207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kntg" event={"ID":"d2e560e2-d355-45c9-90ab-7c6baca64f5b","Type":"ContainerStarted","Data":"a868700b98163dd812fcc7b791fbfe1d6ca91d9f93de1db00d4f6257e0836dab"}
Jul 07 17:41:02.347349 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.347234    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8kntg" podStartSLOduration=3.644788059 podStartE2EDuration="23.347216618s" totalImagesPullingTime="19.702428559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:02.346885002 +0000 UTC m=+23.920832345" watchObservedRunningTime="2026-07-07 17:41:02.347216618 +0000 UTC m=+23.921163960"
Jul 07 17:41:02.458117 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.458061    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-glpm8" podStartSLOduration=12.385269567 podStartE2EDuration="23.458045552s" totalImagesPullingTime="11.072775985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:02.456128595 +0000 UTC m=+24.030075930" watchObservedRunningTime="2026-07-07 17:41:02.458045552 +0000 UTC m=+24.031992894"
Jul 07 17:41:02.458244 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.458154    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fv45b" podStartSLOduration=3.79141848 podStartE2EDuration="23.458147611s" totalImagesPullingTime="19.666729131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:02.427643982 +0000 UTC m=+24.001591324" watchObservedRunningTime="2026-07-07 17:41:02.458147611 +0000 UTC m=+24.032094951"
Jul 07 17:41:02.518047 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.518012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:41:02.518629 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:02.518611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:41:03.091678 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.091644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:03.091866 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:03.091764    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:03.305533 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.305502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" event={"ID":"4f89bf03cbf59db5897f34916fe53bd2","Type":"ContainerStarted","Data":"bf1681edbf3b83db878b0755e67833aebf1e02786488b4d931809bd1555d4f80"}
Jul 07 17:41:03.307263 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.307231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w27zq" event={"ID":"626a6665-283d-4b4e-94ab-d83f79c5d17c","Type":"ContainerStarted","Data":"9e6977e774ace465e86b0609e3839cf60489603e2a6569f3db5a692e9ea0d754"}
Jul 07 17:41:03.308008 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.307848    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:41:03.308413 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.308386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-glpm8"
Jul 07 17:41:03.332060 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.332019    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-144.ec2.internal" podStartSLOduration=23.332003257 podStartE2EDuration="23.332003257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:03.331371001 +0000 UTC m=+24.905318344" watchObservedRunningTime="2026-07-07 17:41:03.332003257 +0000 UTC m=+24.905950598"
Jul 07 17:41:03.471267 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.471243    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 17:41:03.945224 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.945102    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T17:41:03.471260867Z","UUID":"2db9a3c9-fab5-49a8-9f90-3ded8bdfe101","Handler":null,"Name":"","Endpoint":""}
Jul 07 17:41:03.948336 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.948308    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 17:41:03.948472 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:03.948356    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 17:41:04.091511 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:04.091474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:04.091661 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:04.091481    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:04.091661 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:04.091593    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:04.091791 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:04.091706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:04.312205 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:04.312162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl" event={"ID":"f453a5d1-9bee-483a-9191-c73cc2e4eaaa","Type":"ContainerStarted","Data":"5612541a25b3ab545df2bbdbc5eab8d6ec530caa1bb39ad2059d53c9b0ccddae"}
Jul 07 17:41:04.315554 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:04.315533    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:41:04.315953 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:04.315925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"f8169b5f510df53840124fddf4da4158b1ce222bd13d2d08cf5c4acc14315a13"}
Jul 07 17:41:05.091544 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:05.091510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:05.091705 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:05.091642    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:05.320625 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:05.320588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl" event={"ID":"f453a5d1-9bee-483a-9191-c73cc2e4eaaa","Type":"ContainerStarted","Data":"e65f323ed8f05d84217faa4ea34f672b8ca1b98f9af211b612973fa805005d43"}
Jul 07 17:41:05.348998 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:05.348906    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w27zq" podStartSLOduration=6.717526553 podStartE2EDuration="26.348890273s" totalImagesPullingTime="19.63136372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:03.39690285 +0000 UTC m=+24.970850192" watchObservedRunningTime="2026-07-07 17:41:05.348890273 +0000 UTC m=+26.922837615"
Jul 07 17:41:06.091295 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:06.091262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:06.091295 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:06.091290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:06.091539 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:06.091389    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:06.091539 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:06.091523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:07.091571 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.091321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:07.092303 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:07.091685    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:07.328242 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.328188    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:41:07.328595 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.328563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"fcc896b1ff1b64187e3bce1b6383e7bd0b6e9f025fecf32f9568c1740f20b630"}
Jul 07 17:41:07.328880 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.328857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:41:07.328991 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.328922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:41:07.328991 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.328985    2577 scope.go:117] "RemoveContainer" containerID="82da5823a19964d699b770f694e75ba6231060649ff338b8981ae571258ef22c"
Jul 07 17:41:07.330670 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.330644    2577 generic.go:356] "Generic (PLEG): container finished" podID="988895c8-9fa3-482b-97a1-9815c60b8afa" containerID="0a1e26e55203fcc328a9729991f4750a160b6ae808d9344ad2d2c101855d7cf6" exitCode=0
Jul 07 17:41:07.330783 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.330674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerDied","Data":"0a1e26e55203fcc328a9729991f4750a160b6ae808d9344ad2d2c101855d7cf6"}
Jul 07 17:41:07.345591 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.345568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:41:07.346339 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.346322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:41:07.362172 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:07.362131    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x2dtl" podStartSLOduration=6.723470096 podStartE2EDuration="28.362104075s" totalImagesPullingTime="21.638633979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:05.350873618 +0000 UTC m=+26.924820960" watchObservedRunningTime="2026-07-07 17:41:07.362104075 +0000 UTC m=+28.936051423"
Jul 07 17:41:08.090954 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.090770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:08.091187 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.090805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:08.091187 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:08.091049    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:08.091187 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:08.091094    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:08.335816 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.335795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:41:08.336198 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.336125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" event={"ID":"c610d36a-d68f-4a47-90a0-62f695ccace9","Type":"ContainerStarted","Data":"69a798b0e807ee5ffa1ea807e80d3f4f7bbfab7900a587f5ec2919d5bdc1ff96"}
Jul 07 17:41:08.336253 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.336217    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 17:41:08.368748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.368697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2" podStartSLOduration=9.284186528 podStartE2EDuration="29.368682488s" totalImagesPullingTime="20.08449596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:08.366170068 +0000 UTC m=+29.940117411" watchObservedRunningTime="2026-07-07 17:41:08.368682488 +0000 UTC m=+29.942629830"
Jul 07 17:41:08.391418 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.391391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4rd2v"]
Jul 07 17:41:08.391659 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.391523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:08.391659 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:08.391612    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:08.395617 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.395592    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6s6cr"]
Jul 07 17:41:08.395738 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.395726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:08.395859 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:08.395837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:08.396220 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.396196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tnmvh"]
Jul 07 17:41:08.396319 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:08.396273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:08.396380 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:08.396339    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:09.340308 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:09.340269    2577 generic.go:356] "Generic (PLEG): container finished" podID="988895c8-9fa3-482b-97a1-9815c60b8afa" containerID="f685bd3e52d27f51dcfdce5dfd4bc781b43dde97cd5deb97d24e83c66469589c" exitCode=0
Jul 07 17:41:09.340819 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:09.340347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerDied","Data":"f685bd3e52d27f51dcfdce5dfd4bc781b43dde97cd5deb97d24e83c66469589c"}
Jul 07 17:41:09.340819 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:09.340426    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 07 17:41:10.090735 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:10.090710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:10.090930 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:10.090710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:10.090930 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:10.090828    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:10.090930 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:10.090710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:10.091149 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:10.090876    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:10.091149 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:10.090964    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:10.680386 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:10.680350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:41:11.347047 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:11.347010    2577 generic.go:356] "Generic (PLEG): container finished" podID="988895c8-9fa3-482b-97a1-9815c60b8afa" containerID="a9abb34fb5f14a8269b7ddef5b211bd6299ad37be7d3a9dd8f913282d31aa64b" exitCode=0
Jul 07 17:41:11.347245 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:11.347055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerDied","Data":"a9abb34fb5f14a8269b7ddef5b211bd6299ad37be7d3a9dd8f913282d31aa64b"}
Jul 07 17:41:12.091248 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:12.091218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:12.091642 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:12.091223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:12.091642 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.091326    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:12.091642 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.091392    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:12.091642 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:12.091227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:12.091642 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.091458    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:12.770574 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:12.770538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:12.770753 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:12.770589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:12.770753 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:12.770634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:12.770753 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770705    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 17:41:12.770753 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770725    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:41:12.770987 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770759    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 17:41:12.770987 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770779    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 17:41:12.770987 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770780    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs podName:e6523d94-f705-49fb-ac51-83169974230c nodeName:}" failed. No retries permitted until 2026-07-07 17:41:44.770760725 +0000 UTC m=+66.344708043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs") pod "network-metrics-daemon-6s6cr" (UID: "e6523d94-f705-49fb-ac51-83169974230c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 17:41:12.770987 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770791    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wn2cx for pod openshift-network-diagnostics/network-check-target-tnmvh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:41:12.770987 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret podName:1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:44.770790016 +0000 UTC m=+66.344737340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret") pod "global-pull-secret-syncer-4rd2v" (UID: "1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51") : object "kube-system"/"original-pull-secret" not registered
Jul 07 17:41:12.770987 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:12.770826    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx podName:1967f80f-c13c-46b2-91e5-067f3d56691c nodeName:}" failed. No retries permitted until 2026-07-07 17:41:44.770816178 +0000 UTC m=+66.344763512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wn2cx" (UniqueName: "kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx") pod "network-check-target-tnmvh" (UID: "1967f80f-c13c-46b2-91e5-067f3d56691c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 17:41:14.091178 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:14.090945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:14.091639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:14.090950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:14.091639 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:14.091293    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s6cr" podUID="e6523d94-f705-49fb-ac51-83169974230c"
Jul 07 17:41:14.091639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:14.091007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:14.091639 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:14.091322    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tnmvh" podUID="1967f80f-c13c-46b2-91e5-067f3d56691c"
Jul 07 17:41:14.091639 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:14.091461    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rd2v" podUID="1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51"
Jul 07 17:41:15.157060 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.157026    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-144.ec2.internal" event="NodeReady"
Jul 07 17:41:15.157511 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.157203    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 17:41:15.202848 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.202792    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"]
Jul 07 17:41:15.205547 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.205528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.208640 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.208531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 17:41:15.209139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.208647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vx5vk\""
Jul 07 17:41:15.209139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.208696    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 17:41:15.209139 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.208920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 17:41:15.214060 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.214041    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 17:41:15.226221 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.226198    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2bpht"]
Jul 07 17:41:15.228587 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.228555    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h4xxw"]
Jul 07 17:41:15.228781 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.228766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.230240 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.230220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"]
Jul 07 17:41:15.230326 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.230310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.232588 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.232569    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 17:41:15.232682 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.232595    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 17:41:15.232741 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.232707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 17:41:15.233085 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.232828    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 17:41:15.233085 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.232870    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8stmj\""
Jul 07 17:41:15.233085 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.232914    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6c56v\""
Jul 07 17:41:15.233356 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.233343    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 17:41:15.244781 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.244756    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bpht"]
Jul 07 17:41:15.247875 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.247853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4xxw"]
Jul 07 17:41:15.287453 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzmq\" (UniqueName: \"kubernetes.io/projected/15b03e63-d408-4d95-a9e5-60b317778fc3-kube-api-access-5gzmq\") pod \"ingress-canary-h4xxw\" (UID: \"15b03e63-d408-4d95-a9e5-60b317778fc3\") " pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.287639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-image-registry-private-configuration\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-certificates\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-trusted-ca\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-installation-pull-secrets\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-bound-sa-token\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287881 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-ca-trust-extracted\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287881 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqnc7\" (UniqueName: \"kubernetes.io/projected/5e624548-2110-4f89-b179-c30ea2dd9c08-kube-api-access-nqnc7\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.287881 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287881 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxg8\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-kube-api-access-xjxg8\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.287881 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15b03e63-d408-4d95-a9e5-60b317778fc3-cert\") pod \"ingress-canary-h4xxw\" (UID: \"15b03e63-d408-4d95-a9e5-60b317778fc3\") " pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.288072 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e624548-2110-4f89-b179-c30ea2dd9c08-config-volume\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.288072 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e624548-2110-4f89-b179-c30ea2dd9c08-tmp-dir\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.288072 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.287975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.388941 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.388909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-ca-trust-extracted\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.388949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqnc7\" (UniqueName: \"kubernetes.io/projected/5e624548-2110-4f89-b179-c30ea2dd9c08-kube-api-access-nqnc7\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.388976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxg8\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-kube-api-access-xjxg8\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15b03e63-d408-4d95-a9e5-60b317778fc3-cert\") pod \"ingress-canary-h4xxw\" (UID: \"15b03e63-d408-4d95-a9e5-60b317778fc3\") " pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e624548-2110-4f89-b179-c30ea2dd9c08-config-volume\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e624548-2110-4f89-b179-c30ea2dd9c08-tmp-dir\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.389147    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 17:41:15.389163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzmq\" (UniqueName: \"kubernetes.io/projected/15b03e63-d408-4d95-a9e5-60b317778fc3-kube-api-access-5gzmq\") pod \"ingress-canary-h4xxw\" (UID: \"15b03e63-d408-4d95-a9e5-60b317778fc3\") " pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-image-registry-private-configuration\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-certificates\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-trusted-ca\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-installation-pull-secrets\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.389169    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj: secret "image-registry-tls" not found
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-bound-sa-token\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.389382    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls podName:b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d nodeName:}" failed. No retries permitted until 2026-07-07 17:41:15.889356988 +0000 UTC m=+37.463304324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls") pod "image-registry-6dfcf5f8d7-xr5vj" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d") : secret "image-registry-tls" not found
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-ca-trust-extracted\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389481    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e624548-2110-4f89-b179-c30ea2dd9c08-tmp-dir\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.389608 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.389585    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 17:41:15.390102 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.389660    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls podName:5e624548-2110-4f89-b179-c30ea2dd9c08 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:15.88964268 +0000 UTC m=+37.463590008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls") pod "dns-default-2bpht" (UID: "5e624548-2110-4f89-b179-c30ea2dd9c08") : secret "dns-default-metrics-tls" not found
Jul 07 17:41:15.390102 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.389746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e624548-2110-4f89-b179-c30ea2dd9c08-config-volume\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.390102 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.390016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-certificates\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.390575 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.390553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-trusted-ca\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.393722 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.393699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-installation-pull-secrets\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.393841 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.393785    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-image-registry-private-configuration\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.393892 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.393857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15b03e63-d408-4d95-a9e5-60b317778fc3-cert\") pod \"ingress-canary-h4xxw\" (UID: \"15b03e63-d408-4d95-a9e5-60b317778fc3\") " pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.401685 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.401622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqnc7\" (UniqueName: \"kubernetes.io/projected/5e624548-2110-4f89-b179-c30ea2dd9c08-kube-api-access-nqnc7\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.401927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.401902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzmq\" (UniqueName: \"kubernetes.io/projected/15b03e63-d408-4d95-a9e5-60b317778fc3-kube-api-access-5gzmq\") pod \"ingress-canary-h4xxw\" (UID: \"15b03e63-d408-4d95-a9e5-60b317778fc3\") " pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.402320 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.402299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxg8\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-kube-api-access-xjxg8\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.403059 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.403038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-bound-sa-token\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.546921 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.546878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h4xxw"
Jul 07 17:41:15.812432 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.812355    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"]
Jul 07 17:41:15.814279 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.814262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"
Jul 07 17:41:15.819203 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.819179    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 17:41:15.819321 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.819232    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 17:41:15.819553 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.819535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pljhp\""
Jul 07 17:41:15.833327 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.833302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"]
Jul 07 17:41:15.893428 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.893398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpc7\" (UniqueName: \"kubernetes.io/projected/946bb130-666d-4fb0-a145-af56ee5d2e85-kube-api-access-xcpc7\") pod \"migrator-d57d58cc9-4t562\" (UID: \"946bb130-666d-4fb0-a145-af56ee5d2e85\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"
Jul 07 17:41:15.893604 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.893541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:15.893677 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.893618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:15.893742 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.893724    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 17:41:15.893789 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.893749    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj: secret "image-registry-tls" not found
Jul 07 17:41:15.893789 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.893729    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 17:41:15.893880 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.893815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls podName:b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d nodeName:}" failed. No retries permitted until 2026-07-07 17:41:16.893793415 +0000 UTC m=+38.467740737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls") pod "image-registry-6dfcf5f8d7-xr5vj" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d") : secret "image-registry-tls" not found
Jul 07 17:41:15.893880 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:15.893834    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls podName:5e624548-2110-4f89-b179-c30ea2dd9c08 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:16.893824046 +0000 UTC m=+38.467771370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls") pod "dns-default-2bpht" (UID: "5e624548-2110-4f89-b179-c30ea2dd9c08") : secret "dns-default-metrics-tls" not found
Jul 07 17:41:15.995072 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:15.995031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpc7\" (UniqueName: \"kubernetes.io/projected/946bb130-666d-4fb0-a145-af56ee5d2e85-kube-api-access-xcpc7\") pod \"migrator-d57d58cc9-4t562\" (UID: \"946bb130-666d-4fb0-a145-af56ee5d2e85\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"
Jul 07 17:41:16.022331 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.022302    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpc7\" (UniqueName: \"kubernetes.io/projected/946bb130-666d-4fb0-a145-af56ee5d2e85-kube-api-access-xcpc7\") pod \"migrator-d57d58cc9-4t562\" (UID: \"946bb130-666d-4fb0-a145-af56ee5d2e85\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"
Jul 07 17:41:16.091327 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.091239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:16.091493 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.091239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:16.091493 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.091254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:16.094558 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.094529    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2s8zn"]
Jul 07 17:41:16.095045 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.095021    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pvl4h\""
Jul 07 17:41:16.095289 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.095273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 17:41:16.095376 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.095310    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 17:41:16.095431 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.095385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 17:41:16.095431 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.095409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 17:41:16.095540 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.095530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2hwnn\""
Jul 07 17:41:16.096739 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.096723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.105644 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.105624    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 17:41:16.105739 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.105703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 17:41:16.106399 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.106219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 17:41:16.106399 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.106243    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hzvmm\""
Jul 07 17:41:16.106399 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.106321    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 17:41:16.121679 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.121661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2s8zn"]
Jul 07 17:41:16.123684 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.123664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"
Jul 07 17:41:16.195923 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.195890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60c44b63-0ee5-4f79-94c3-b2538bfb6088-data-volume\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.196324 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.195930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.196324 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.195962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60c44b63-0ee5-4f79-94c3-b2538bfb6088-crio-socket\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.196324 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.196012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.196324 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.196098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhqm\" (UniqueName: \"kubernetes.io/projected/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-api-access-bfhqm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297314 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60c44b63-0ee5-4f79-94c3-b2538bfb6088-data-volume\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297314 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297549 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60c44b63-0ee5-4f79-94c3-b2538bfb6088-crio-socket\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297549 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297549 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhqm\" (UniqueName: \"kubernetes.io/projected/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-api-access-bfhqm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297549 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.297503    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:16.297690 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.297512    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 17:41:16.297690 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60c44b63-0ee5-4f79-94c3-b2538bfb6088-crio-socket\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.297690 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.297662    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:16.797634989 +0000 UTC m=+38.371582313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:16.297690 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.297677    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:16.797670559 +0000 UTC m=+38.371617878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : configmap "kube-rbac-proxy" not found
Jul 07 17:41:16.297893 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.297707    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60c44b63-0ee5-4f79-94c3-b2538bfb6088-data-volume\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.307454 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.307420    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhqm\" (UniqueName: \"kubernetes.io/projected/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-api-access-bfhqm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.476556 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.476384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562"]
Jul 07 17:41:16.479628 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:16.479572    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946bb130_666d_4fb0_a145_af56ee5d2e85.slice/crio-9d24387856c327b7265e475941c0f73bc6aadc7c631fea1555580e6037dcfcb6 WatchSource:0}: Error finding container 9d24387856c327b7265e475941c0f73bc6aadc7c631fea1555580e6037dcfcb6: Status 404 returned error can't find the container with id 9d24387856c327b7265e475941c0f73bc6aadc7c631fea1555580e6037dcfcb6
Jul 07 17:41:16.480087 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.480063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h4xxw"]
Jul 07 17:41:16.482180 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:16.482157    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b03e63_d408_4d95_a9e5_60b317778fc3.slice/crio-d706bd8061450a221953f7e84b9601f17cddeab8282f6ec28f3bf4f08e4cb3f5 WatchSource:0}: Error finding container d706bd8061450a221953f7e84b9601f17cddeab8282f6ec28f3bf4f08e4cb3f5: Status 404 returned error can't find the container with id d706bd8061450a221953f7e84b9601f17cddeab8282f6ec28f3bf4f08e4cb3f5
Jul 07 17:41:16.804170 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.804132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.804383 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.804216    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 17:41:16.804383 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.804240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:16.804383 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.804292    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:17.804270932 +0000 UTC m=+39.378218251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : configmap "kube-rbac-proxy" not found
Jul 07 17:41:16.804383 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.804335    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:16.804383 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.804379    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:17.804367428 +0000 UTC m=+39.378314746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:16.904820 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.904783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:16.904996 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:16.904873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:16.904996 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.904931    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 17:41:16.904996 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.904956    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj: secret "image-registry-tls" not found
Jul 07 17:41:16.905198 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.905025    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls podName:b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d nodeName:}" failed. No retries permitted until 2026-07-07 17:41:18.905002248 +0000 UTC m=+40.478949600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls") pod "image-registry-6dfcf5f8d7-xr5vj" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d") : secret "image-registry-tls" not found
Jul 07 17:41:16.905198 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.905030    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 17:41:16.905198 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:16.905092    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls podName:5e624548-2110-4f89-b179-c30ea2dd9c08 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:18.905074405 +0000 UTC m=+40.479021741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls") pod "dns-default-2bpht" (UID: "5e624548-2110-4f89-b179-c30ea2dd9c08") : secret "dns-default-metrics-tls" not found
Jul 07 17:41:17.365078 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:17.365041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562" event={"ID":"946bb130-666d-4fb0-a145-af56ee5d2e85","Type":"ContainerStarted","Data":"9d24387856c327b7265e475941c0f73bc6aadc7c631fea1555580e6037dcfcb6"}
Jul 07 17:41:17.368125 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:17.368083    2577 generic.go:356] "Generic (PLEG): container finished" podID="988895c8-9fa3-482b-97a1-9815c60b8afa" containerID="05da98177097f9f8a95e194d38b402dd696651843784a5367d41c40e52ed6405" exitCode=0
Jul 07 17:41:17.368248 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:17.368182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerDied","Data":"05da98177097f9f8a95e194d38b402dd696651843784a5367d41c40e52ed6405"}
Jul 07 17:41:17.369763 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:17.369733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4xxw" event={"ID":"15b03e63-d408-4d95-a9e5-60b317778fc3","Type":"ContainerStarted","Data":"d706bd8061450a221953f7e84b9601f17cddeab8282f6ec28f3bf4f08e4cb3f5"}
Jul 07 17:41:17.813260 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:17.813222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:17.813441 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:17.813355    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 17:41:17.813441 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:17.813391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:17.813441 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:17.813416    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:19.813400741 +0000 UTC m=+41.387348061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : configmap "kube-rbac-proxy" not found
Jul 07 17:41:17.813601 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:17.813491    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:17.813601 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:17.813547    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:19.813532779 +0000 UTC m=+41.387480102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:18.243380 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.243307    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fv45b_0e039226-8e03-42d0-b084-f552e7b4c2b5/dns-node-resolver/0.log"
Jul 07 17:41:18.325972 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.325865    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mspvh"]
Jul 07 17:41:18.345553 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.345518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.345891 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.345865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mspvh"]
Jul 07 17:41:18.350070 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.349750    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 17:41:18.350070 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.349908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 17:41:18.350070 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.350043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 17:41:18.350516 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.350457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xx66s\""
Jul 07 17:41:18.350516 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.350491    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 17:41:18.374875 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.374602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562" event={"ID":"946bb130-666d-4fb0-a145-af56ee5d2e85","Type":"ContainerStarted","Data":"3abde5d212f703a79e0aa0e8c78112456ae60246dbcd84a0765fd52be9161cfc"}
Jul 07 17:41:18.374875 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.374667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562" event={"ID":"946bb130-666d-4fb0-a145-af56ee5d2e85","Type":"ContainerStarted","Data":"a6025b3251f7cd470eb8af7dc24c4306917592d219ebcf05b922528816d55799"}
Jul 07 17:41:18.378209 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.378174    2577 generic.go:356] "Generic (PLEG): container finished" podID="988895c8-9fa3-482b-97a1-9815c60b8afa" containerID="ccffe4ca346436397206d391babee6c742bfd28729d0b96de26f8159d159ba13" exitCode=0
Jul 07 17:41:18.378314 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.378226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerDied","Data":"ccffe4ca346436397206d391babee6c742bfd28729d0b96de26f8159d159ba13"}
Jul 07 17:41:18.410647 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.410600    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-4t562" podStartSLOduration=2.065967913 podStartE2EDuration="3.410581897s" totalImagesPullingTime="1.344613984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:18.410579118 +0000 UTC m=+39.984526462" watchObservedRunningTime="2026-07-07 17:41:18.410581897 +0000 UTC m=+39.984529232"
Jul 07 17:41:18.419179 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.419148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00b13900-794e-48aa-9aa4-6c1bb6c65a07-signing-cabundle\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.419347 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.419200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zqr\" (UniqueName: \"kubernetes.io/projected/00b13900-794e-48aa-9aa4-6c1bb6c65a07-kube-api-access-98zqr\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.419347 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.419251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00b13900-794e-48aa-9aa4-6c1bb6c65a07-signing-key\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.519937 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.519907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00b13900-794e-48aa-9aa4-6c1bb6c65a07-signing-cabundle\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.520148 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.519958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zqr\" (UniqueName: \"kubernetes.io/projected/00b13900-794e-48aa-9aa4-6c1bb6c65a07-kube-api-access-98zqr\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.520148 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.519999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00b13900-794e-48aa-9aa4-6c1bb6c65a07-signing-key\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.520695 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.520668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/00b13900-794e-48aa-9aa4-6c1bb6c65a07-signing-cabundle\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.524318 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.524295    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/00b13900-794e-48aa-9aa4-6c1bb6c65a07-signing-key\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.532678 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.532655    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zqr\" (UniqueName: \"kubernetes.io/projected/00b13900-794e-48aa-9aa4-6c1bb6c65a07-kube-api-access-98zqr\") pod \"service-ca-758f599c5-mspvh\" (UID: \"00b13900-794e-48aa-9aa4-6c1bb6c65a07\") " pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.657828 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.657797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-mspvh"
Jul 07 17:41:18.876667 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.876635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mspvh"]
Jul 07 17:41:18.879804 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:18.879780    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b13900_794e_48aa_9aa4_6c1bb6c65a07.slice/crio-53434a9d1e849d09a2604e4c79d2cd6b9cf6d979f3ecdc70b8fe3759bd58675b WatchSource:0}: Error finding container 53434a9d1e849d09a2604e4c79d2cd6b9cf6d979f3ecdc70b8fe3759bd58675b: Status 404 returned error can't find the container with id 53434a9d1e849d09a2604e4c79d2cd6b9cf6d979f3ecdc70b8fe3759bd58675b
Jul 07 17:41:18.922514 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.922482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:18.922740 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:18.922536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:18.922740 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:18.922663    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 17:41:18.922740 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:18.922693    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj: secret "image-registry-tls" not found
Jul 07 17:41:18.922899 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:18.922749    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls podName:b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d nodeName:}" failed. No retries permitted until 2026-07-07 17:41:22.922728761 +0000 UTC m=+44.496676094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls") pod "image-registry-6dfcf5f8d7-xr5vj" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d") : secret "image-registry-tls" not found
Jul 07 17:41:18.922899 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:18.922666    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 17:41:18.922899 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:18.922794    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls podName:5e624548-2110-4f89-b179-c30ea2dd9c08 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:22.922782876 +0000 UTC m=+44.496730195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls") pod "dns-default-2bpht" (UID: "5e624548-2110-4f89-b179-c30ea2dd9c08") : secret "dns-default-metrics-tls" not found
Jul 07 17:41:19.245975 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.245930    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kntg_d2e560e2-d355-45c9-90ab-7c6baca64f5b/node-ca/0.log"
Jul 07 17:41:19.384672 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.384586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" event={"ID":"988895c8-9fa3-482b-97a1-9815c60b8afa","Type":"ContainerStarted","Data":"fa9fa37bb620e4b259b812c811c2655a087a084652c0adf3b0d8ceaf6fad0b2a"}
Jul 07 17:41:19.386224 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.386194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h4xxw" event={"ID":"15b03e63-d408-4d95-a9e5-60b317778fc3","Type":"ContainerStarted","Data":"d0af84b8ccda0e3913e54508fca5126887f9c8f430b733d94e6ac2c87af5c8ab"}
Jul 07 17:41:19.387402 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.387380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-mspvh" event={"ID":"00b13900-794e-48aa-9aa4-6c1bb6c65a07","Type":"ContainerStarted","Data":"53434a9d1e849d09a2604e4c79d2cd6b9cf6d979f3ecdc70b8fe3759bd58675b"}
Jul 07 17:41:19.421101 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.421056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pd4lj" podStartSLOduration=9.72693676 podStartE2EDuration="40.421043054s" totalImagesPullingTime="30.694106294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:19.420663247 +0000 UTC m=+40.994610587" watchObservedRunningTime="2026-07-07 17:41:19.421043054 +0000 UTC m=+40.994990395"
Jul 07 17:41:19.443583 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.443445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h4xxw" podStartSLOduration=1.803643221 podStartE2EDuration="4.443426462s" totalImagesPullingTime="2.639783241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:19.443068722 +0000 UTC m=+41.017016063" watchObservedRunningTime="2026-07-07 17:41:19.443426462 +0000 UTC m=+41.017373805"
Jul 07 17:41:19.828516 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.828482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:19.828711 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:19.828537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:19.828711 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:19.828667    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:19.828711 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:19.828682    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 17:41:19.828857 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:19.828748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:23.82872848 +0000 UTC m=+45.402675804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:19.828857 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:19.828767    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:23.828757914 +0000 UTC m=+45.402705233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : configmap "kube-rbac-proxy" not found
Jul 07 17:41:20.041678 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:20.041639    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4xxw_15b03e63-d408-4d95-a9e5-60b317778fc3/serve-healthcheck-canary/0.log"
Jul 07 17:41:20.439362 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:20.439331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4t562_946bb130-666d-4fb0-a145-af56ee5d2e85/migrator/0.log"
Jul 07 17:41:20.638558 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:20.638530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4t562_946bb130-666d-4fb0-a145-af56ee5d2e85/graceful-termination/0.log"
Jul 07 17:41:22.396761 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:22.396719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-mspvh" event={"ID":"00b13900-794e-48aa-9aa4-6c1bb6c65a07","Type":"ContainerStarted","Data":"4652d53c22e34151b392f98dba3fcfc1a13978ba9f6fb82a9834e98495ccddbd"}
Jul 07 17:41:22.414986 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:22.414931    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-mspvh" podStartSLOduration=1.962255185 podStartE2EDuration="4.414916329s" totalImagesPullingTime="2.452661144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:22.413507101 +0000 UTC m=+43.987454442" watchObservedRunningTime="2026-07-07 17:41:22.414916329 +0000 UTC m=+43.988863669"
Jul 07 17:41:22.958030 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:22.957987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:22.958205 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:22.958057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:22.958205 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:22.958192    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 17:41:22.958273 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:22.958206    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 17:41:22.958273 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:22.958217    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj: secret "image-registry-tls" not found
Jul 07 17:41:22.958273 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:22.958269    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls podName:5e624548-2110-4f89-b179-c30ea2dd9c08 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:30.958254143 +0000 UTC m=+52.532201462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls") pod "dns-default-2bpht" (UID: "5e624548-2110-4f89-b179-c30ea2dd9c08") : secret "dns-default-metrics-tls" not found
Jul 07 17:41:22.958389 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:22.958284    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls podName:b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d nodeName:}" failed. No retries permitted until 2026-07-07 17:41:30.958277909 +0000 UTC m=+52.532225227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls") pod "image-registry-6dfcf5f8d7-xr5vj" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d") : secret "image-registry-tls" not found
Jul 07 17:41:23.865991 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:23.865951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:23.866488 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:23.866014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:23.866488 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:23.866135    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:23.866488 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:23.866206    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:31.866185608 +0000 UTC m=+53.440132933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:23.866488 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:23.866208    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 17:41:23.866488 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:23.866283    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:31.866266483 +0000 UTC m=+53.440213808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : configmap "kube-rbac-proxy" not found
Jul 07 17:41:31.025158 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.025101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:31.025590 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.025287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:31.027471 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.027437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e624548-2110-4f89-b179-c30ea2dd9c08-metrics-tls\") pod \"dns-default-2bpht\" (UID: \"5e624548-2110-4f89-b179-c30ea2dd9c08\") " pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:31.027580 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.027474    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"image-registry-6dfcf5f8d7-xr5vj\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") " pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:31.117720 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.117685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:31.141027 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.140994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:31.270008 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.269974    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"]
Jul 07 17:41:31.273747 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:31.273714    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ebfaf9_4924_4b1f_93e2_9ba8c98fb76d.slice/crio-99a5a6119b24e0732df0dc54ef5396124bc4134b19b64208560dabf3e1c3da09 WatchSource:0}: Error finding container 99a5a6119b24e0732df0dc54ef5396124bc4134b19b64208560dabf3e1c3da09: Status 404 returned error can't find the container with id 99a5a6119b24e0732df0dc54ef5396124bc4134b19b64208560dabf3e1c3da09
Jul 07 17:41:31.289064 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.289043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bpht"]
Jul 07 17:41:31.292062 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:31.292038    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e624548_2110_4f89_b179_c30ea2dd9c08.slice/crio-37ed8ce233732964306dcf7167a9d888ae3b974384468666b80ac030cf29b1b6 WatchSource:0}: Error finding container 37ed8ce233732964306dcf7167a9d888ae3b974384468666b80ac030cf29b1b6: Status 404 returned error can't find the container with id 37ed8ce233732964306dcf7167a9d888ae3b974384468666b80ac030cf29b1b6
Jul 07 17:41:31.421023 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.420988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bpht" event={"ID":"5e624548-2110-4f89-b179-c30ea2dd9c08","Type":"ContainerStarted","Data":"37ed8ce233732964306dcf7167a9d888ae3b974384468666b80ac030cf29b1b6"}
Jul 07 17:41:31.422603 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.422579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj" event={"ID":"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d","Type":"ContainerStarted","Data":"bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab"}
Jul 07 17:41:31.422719 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.422622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj" event={"ID":"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d","Type":"ContainerStarted","Data":"99a5a6119b24e0732df0dc54ef5396124bc4134b19b64208560dabf3e1c3da09"}
Jul 07 17:41:31.422719 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.422650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:31.443801 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.443752    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj" podStartSLOduration=22.443737456 podStartE2EDuration="22.443737456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:31.442833206 +0000 UTC m=+53.016780547" watchObservedRunningTime="2026-07-07 17:41:31.443737456 +0000 UTC m=+53.017684806"
Jul 07 17:41:31.934539 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.934499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:31.934727 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.934626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:31.934793 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:31.934738    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:31.934847 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:41:31.934799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls podName:60c44b63-0ee5-4f79-94c3-b2538bfb6088 nodeName:}" failed. No retries permitted until 2026-07-07 17:41:47.934783625 +0000 UTC m=+69.508730970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2s8zn" (UID: "60c44b63-0ee5-4f79-94c3-b2538bfb6088") : secret "insights-runtime-extractor-tls" not found
Jul 07 17:41:31.935163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:31.935130    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60c44b63-0ee5-4f79-94c3-b2538bfb6088-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:34.434908 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:34.434875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bpht" event={"ID":"5e624548-2110-4f89-b179-c30ea2dd9c08","Type":"ContainerStarted","Data":"a13e4b84fb000e8752b778eaf9de93191503b61546516e8729ac94e8f958b06f"}
Jul 07 17:41:34.435394 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:34.434933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bpht" event={"ID":"5e624548-2110-4f89-b179-c30ea2dd9c08","Type":"ContainerStarted","Data":"e4e9fabc669a92da6e42603d9728f90675a0f2905a9e380a1b894bdccc7f5917"}
Jul 07 17:41:34.435394 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:34.435012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:34.467215 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:34.467166    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2bpht" podStartSLOduration=17.071929777 podStartE2EDuration="19.467152324s" totalImagesPullingTime="2.395222547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:34.466229973 +0000 UTC m=+56.040177333" watchObservedRunningTime="2026-07-07 17:41:34.467152324 +0000 UTC m=+56.041099664"
Jul 07 17:41:40.699089 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:40.699059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bw4c2"
Jul 07 17:41:43.721264 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.721234    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"]
Jul 07 17:41:43.724481 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.724460    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"]
Jul 07 17:41:43.724632 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.724598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.726176 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.726157    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"]
Jul 07 17:41:43.726389 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.726369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:43.728226 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.728202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.730160 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.730097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 17:41:43.730477 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.730455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 07 17:41:43.730477 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.730475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-smdjb\""
Jul 07 17:41:43.730658 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.730461    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 17:41:43.731731 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.731564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 07 17:41:43.731812 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.731761    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 17:41:43.732015 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.731993    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 17:41:43.732135 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.732049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-78w55\""
Jul 07 17:41:43.732135 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.732092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 17:41:43.732296 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.732239    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 17:41:43.732404 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.732377    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 17:41:43.732502 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.732392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 17:41:43.754621 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.754598    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"]
Jul 07 17:41:43.758923 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.758903    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"]
Jul 07 17:41:43.774611 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.774584    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"]
Jul 07 17:41:43.810965 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.810930    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"]
Jul 07 17:41:43.831573 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a44a5ab6-5c81-4e25-8e12-4e78eaa1624a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-gkbmq\" (UID: \"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.831573 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jvs\" (UniqueName: \"kubernetes.io/projected/4814829b-4745-4904-8029-ad1c7dd0f4f4-kube-api-access-h9jvs\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.831791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-hub\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.831791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814829b-4745-4904-8029-ad1c7dd0f4f4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.831791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/184c72f0-6c3e-47a6-ac34-f89d613546ae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56968984f8-2btr9\" (UID: \"184c72f0-6c3e-47a6-ac34-f89d613546ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:43.831791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-ca\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.831955 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a44a5ab6-5c81-4e25-8e12-4e78eaa1624a-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-gkbmq\" (UID: \"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.831955 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.831955 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.831955 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.831936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcr5s\" (UniqueName: \"kubernetes.io/projected/184c72f0-6c3e-47a6-ac34-f89d613546ae-kube-api-access-vcr5s\") pod \"managed-serviceaccount-addon-agent-56968984f8-2btr9\" (UID: \"184c72f0-6c3e-47a6-ac34-f89d613546ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:43.879421 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.879387    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79768b874c-tskt2"]
Jul 07 17:41:43.881321 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.881302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:43.911033 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.911008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79768b874c-tskt2"]
Jul 07 17:41:43.932665 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-hub\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.932784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814829b-4745-4904-8029-ad1c7dd0f4f4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.932784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/184c72f0-6c3e-47a6-ac34-f89d613546ae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56968984f8-2btr9\" (UID: \"184c72f0-6c3e-47a6-ac34-f89d613546ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:43.932784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-ca\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.932893 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a44a5ab6-5c81-4e25-8e12-4e78eaa1624a-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-gkbmq\" (UID: \"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.932893 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.932987 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.932987 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcr5s\" (UniqueName: \"kubernetes.io/projected/184c72f0-6c3e-47a6-ac34-f89d613546ae-kube-api-access-vcr5s\") pod \"managed-serviceaccount-addon-agent-56968984f8-2btr9\" (UID: \"184c72f0-6c3e-47a6-ac34-f89d613546ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:43.932987 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a44a5ab6-5c81-4e25-8e12-4e78eaa1624a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-gkbmq\" (UID: \"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.933232 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.932998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jvs\" (UniqueName: \"kubernetes.io/projected/4814829b-4745-4904-8029-ad1c7dd0f4f4-kube-api-access-h9jvs\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.933467 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.933446    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814829b-4745-4904-8029-ad1c7dd0f4f4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.933549 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.933524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a44a5ab6-5c81-4e25-8e12-4e78eaa1624a-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-gkbmq\" (UID: \"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.935702 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.935676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-ca\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.935807 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.935701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-hub\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.935930 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.935908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/184c72f0-6c3e-47a6-ac34-f89d613546ae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56968984f8-2btr9\" (UID: \"184c72f0-6c3e-47a6-ac34-f89d613546ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:43.936025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.936009    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.936131 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.936096    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4814829b-4745-4904-8029-ad1c7dd0f4f4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.936179 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.936146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a44a5ab6-5c81-4e25-8e12-4e78eaa1624a-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-gkbmq\" (UID: \"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:43.956045 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.956022    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jvs\" (UniqueName: \"kubernetes.io/projected/4814829b-4745-4904-8029-ad1c7dd0f4f4-kube-api-access-h9jvs\") pod \"cluster-proxy-proxy-agent-5878976f55-2r529\" (UID: \"4814829b-4745-4904-8029-ad1c7dd0f4f4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:43.956947 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:43.956924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcr5s\" (UniqueName: \"kubernetes.io/projected/184c72f0-6c3e-47a6-ac34-f89d613546ae-kube-api-access-vcr5s\") pod \"managed-serviceaccount-addon-agent-56968984f8-2btr9\" (UID: \"184c72f0-6c3e-47a6-ac34-f89d613546ae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:44.034020 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.033988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-trusted-ca\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034243 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-installation-pull-secrets\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034243 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-bound-sa-token\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034243 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-registry-tls\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034243 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-image-registry-private-configuration\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034243 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-ca-trust-extracted\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034432 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-registry-certificates\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034432 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxss\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-kube-api-access-qxxss\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.034838 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.034818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"
Jul 07 17:41:44.060817 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.060785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"
Jul 07 17:41:44.064959 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.064933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"
Jul 07 17:41:44.136460 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-trusted-ca\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136562 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-installation-pull-secrets\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136562 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-bound-sa-token\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136633 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-registry-tls\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136633 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-image-registry-private-configuration\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136695 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-ca-trust-extracted\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136695 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-registry-certificates\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.136774 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.136706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxss\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-kube-api-access-qxxss\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.137301 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.137275    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-ca-trust-extracted\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.138309 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.138128    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-registry-certificates\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.138922 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.138885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-trusted-ca\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.139895 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.139877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-image-registry-private-configuration\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.139958 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.139940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-registry-tls\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.140783 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.140764    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-installation-pull-secrets\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.168958 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.168885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-bound-sa-token\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.185946 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.185915    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxss\" (UniqueName: \"kubernetes.io/projected/7ef7e83f-6f3a-4a43-934e-b45dc796aeda-kube-api-access-qxxss\") pod \"image-registry-79768b874c-tskt2\" (UID: \"7ef7e83f-6f3a-4a43-934e-b45dc796aeda\") " pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.196729 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.196699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:44.227242 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.227202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq"]
Jul 07 17:41:44.229046 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:44.229023    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44a5ab6_5c81_4e25_8e12_4e78eaa1624a.slice/crio-c23230651fa11fe67e3090611a04d515ab3ae1b2f7b05eb5eb0ad254ac633e93 WatchSource:0}: Error finding container c23230651fa11fe67e3090611a04d515ab3ae1b2f7b05eb5eb0ad254ac633e93: Status 404 returned error can't find the container with id c23230651fa11fe67e3090611a04d515ab3ae1b2f7b05eb5eb0ad254ac633e93
Jul 07 17:41:44.255559 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:44.255415    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod184c72f0_6c3e_47a6_ac34_f89d613546ae.slice/crio-5afe700ecc757644dc50e68c33ee37246a9cda451dec7cf558550f738fa1ccee WatchSource:0}: Error finding container 5afe700ecc757644dc50e68c33ee37246a9cda451dec7cf558550f738fa1ccee: Status 404 returned error can't find the container with id 5afe700ecc757644dc50e68c33ee37246a9cda451dec7cf558550f738fa1ccee
Jul 07 17:41:44.265123 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.265083    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9"]
Jul 07 17:41:44.299998 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.299933    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529"]
Jul 07 17:41:44.302666 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:44.302637    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4814829b_4745_4904_8029_ad1c7dd0f4f4.slice/crio-ed47cc12cc969624a0930aa50c8116129016f47db746f93e511e60bfad5f51c1 WatchSource:0}: Error finding container ed47cc12cc969624a0930aa50c8116129016f47db746f93e511e60bfad5f51c1: Status 404 returned error can't find the container with id ed47cc12cc969624a0930aa50c8116129016f47db746f93e511e60bfad5f51c1
Jul 07 17:41:44.418484 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.418446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79768b874c-tskt2"]
Jul 07 17:41:44.424222 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:44.424198    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef7e83f_6f3a_4a43_934e_b45dc796aeda.slice/crio-c4a1181cf25c1510e44b8a1c9339fa100a3694515f21a680dea4b9bb7e99559e WatchSource:0}: Error finding container c4a1181cf25c1510e44b8a1c9339fa100a3694515f21a680dea4b9bb7e99559e: Status 404 returned error can't find the container with id c4a1181cf25c1510e44b8a1c9339fa100a3694515f21a680dea4b9bb7e99559e
Jul 07 17:41:44.440674 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.440652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2bpht"
Jul 07 17:41:44.468071 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.468046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79768b874c-tskt2" event={"ID":"7ef7e83f-6f3a-4a43-934e-b45dc796aeda","Type":"ContainerStarted","Data":"c4a1181cf25c1510e44b8a1c9339fa100a3694515f21a680dea4b9bb7e99559e"}
Jul 07 17:41:44.469146 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.469122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9" event={"ID":"184c72f0-6c3e-47a6-ac34-f89d613546ae","Type":"ContainerStarted","Data":"5afe700ecc757644dc50e68c33ee37246a9cda451dec7cf558550f738fa1ccee"}
Jul 07 17:41:44.470213 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.470186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529" event={"ID":"4814829b-4745-4904-8029-ad1c7dd0f4f4","Type":"ContainerStarted","Data":"ed47cc12cc969624a0930aa50c8116129016f47db746f93e511e60bfad5f51c1"}
Jul 07 17:41:44.471216 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.471197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq" event={"ID":"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a","Type":"ContainerStarted","Data":"c23230651fa11fe67e3090611a04d515ab3ae1b2f7b05eb5eb0ad254ac633e93"}
Jul 07 17:41:44.843118 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.843056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:44.843586 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.843151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:44.843586 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.843226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:44.845561 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.845515    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 17:41:44.846229 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.846200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 17:41:44.846229 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.846205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 17:41:44.856149 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.856123    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6523d94-f705-49fb-ac51-83169974230c-metrics-certs\") pod \"network-metrics-daemon-6s6cr\" (UID: \"e6523d94-f705-49fb-ac51-83169974230c\") " pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:44.856149 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.856142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51-original-pull-secret\") pod \"global-pull-secret-syncer-4rd2v\" (UID: \"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51\") " pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:44.857775 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.857724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 17:41:44.866431 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.866410    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2cx\" (UniqueName: \"kubernetes.io/projected/1967f80f-c13c-46b2-91e5-067f3d56691c-kube-api-access-wn2cx\") pod \"network-check-target-tnmvh\" (UID: \"1967f80f-c13c-46b2-91e5-067f3d56691c\") " pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:44.903196 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.903166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rd2v"
Jul 07 17:41:44.913358 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.913231    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2hwnn\""
Jul 07 17:41:44.920388 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.920254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s6cr"
Jul 07 17:41:44.921283 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.921260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pvl4h\""
Jul 07 17:41:44.927328 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:44.927270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:45.142812 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.142780    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4rd2v"]
Jul 07 17:41:45.174772 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.174370    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6s6cr"]
Jul 07 17:41:45.179526 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.179500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tnmvh"]
Jul 07 17:41:45.184184 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:45.184156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6523d94_f705_49fb_ac51_83169974230c.slice/crio-5e58efba79297ce3b4cc1ec46052a2207f48226b1a4a561eca836956e5f99004 WatchSource:0}: Error finding container 5e58efba79297ce3b4cc1ec46052a2207f48226b1a4a561eca836956e5f99004: Status 404 returned error can't find the container with id 5e58efba79297ce3b4cc1ec46052a2207f48226b1a4a561eca836956e5f99004
Jul 07 17:41:45.185760 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:41:45.185737    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1967f80f_c13c_46b2_91e5_067f3d56691c.slice/crio-eccdce44d36d9cb714e0e33ea152a8423f1e3a4b8ce3077c72aa45ac19010f3d WatchSource:0}: Error finding container eccdce44d36d9cb714e0e33ea152a8423f1e3a4b8ce3077c72aa45ac19010f3d: Status 404 returned error can't find the container with id eccdce44d36d9cb714e0e33ea152a8423f1e3a4b8ce3077c72aa45ac19010f3d
Jul 07 17:41:45.480635 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.480490    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tnmvh" event={"ID":"1967f80f-c13c-46b2-91e5-067f3d56691c","Type":"ContainerStarted","Data":"eccdce44d36d9cb714e0e33ea152a8423f1e3a4b8ce3077c72aa45ac19010f3d"}
Jul 07 17:41:45.482956 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.482869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s6cr" event={"ID":"e6523d94-f705-49fb-ac51-83169974230c","Type":"ContainerStarted","Data":"5e58efba79297ce3b4cc1ec46052a2207f48226b1a4a561eca836956e5f99004"}
Jul 07 17:41:45.487086 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.487036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79768b874c-tskt2" event={"ID":"7ef7e83f-6f3a-4a43-934e-b45dc796aeda","Type":"ContainerStarted","Data":"be61df91b2b16bf7fc4b1fc63011e49843d3d9e00de7d45d20a5bd51327e8126"}
Jul 07 17:41:45.487446 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.487240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:41:45.489980 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.489952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4rd2v" event={"ID":"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51","Type":"ContainerStarted","Data":"6192e3b074a0ad3f770191e5bbe01be4d0190816aee16710039f2ef342cc3c9c"}
Jul 07 17:41:45.512634 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:45.512583    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79768b874c-tskt2" podStartSLOduration=2.512565388 podStartE2EDuration="2.512565388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:45.512198794 +0000 UTC m=+67.086146136" watchObservedRunningTime="2026-07-07 17:41:45.512565388 +0000 UTC m=+67.086512727"
Jul 07 17:41:46.498247 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:46.498213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq" event={"ID":"a44a5ab6-5c81-4e25-8e12-4e78eaa1624a","Type":"ContainerStarted","Data":"b7b08a4f74c43945a8d70ebd5d1c6e76ab7a6cd8e6e5fd85439214e8fd8ec5f1"}
Jul 07 17:41:46.518397 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:46.518345    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-gkbmq" podStartSLOduration=1.776002963 podStartE2EDuration="3.518326227s" totalImagesPullingTime="1.742323264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:46.516922184 +0000 UTC m=+68.090869526" watchObservedRunningTime="2026-07-07 17:41:46.518326227 +0000 UTC m=+68.092273570"
Jul 07 17:41:47.974256 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:47.974215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:47.977290 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:47.977238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60c44b63-0ee5-4f79-94c3-b2538bfb6088-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2s8zn\" (UID: \"60c44b63-0ee5-4f79-94c3-b2538bfb6088\") " pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:48.225287 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:48.225152    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hzvmm\""
Jul 07 17:41:48.233325 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:48.233249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2s8zn"
Jul 07 17:41:52.846986 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:52.846874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2s8zn"]
Jul 07 17:41:53.527522 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.526883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9" event={"ID":"184c72f0-6c3e-47a6-ac34-f89d613546ae","Type":"ContainerStarted","Data":"56f0dc4aa5c6a2fe70efba78ff023e49434ca86ffd33445f2be6acd70eea1fb5"}
Jul 07 17:41:53.529549 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.529514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4rd2v" event={"ID":"1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51","Type":"ContainerStarted","Data":"e91f183be0b14fe9faec50ceb0d649f643d421136a045363f58e6c9736bfecf4"}
Jul 07 17:41:53.533929 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.533897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529" event={"ID":"4814829b-4745-4904-8029-ad1c7dd0f4f4","Type":"ContainerStarted","Data":"aea1e285d5a97342ffc6cc5a77e3037fff830298f077db49ee80d850e2cffd1f"}
Jul 07 17:41:53.534081 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.533945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529" event={"ID":"4814829b-4745-4904-8029-ad1c7dd0f4f4","Type":"ContainerStarted","Data":"ec20030119c8cdded18efcb65f726b41fe277cca6c550427a8f413e30cee3bdb"}
Jul 07 17:41:53.534081 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.533960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529" event={"ID":"4814829b-4745-4904-8029-ad1c7dd0f4f4","Type":"ContainerStarted","Data":"53675c90f18fb5d24c195470a1b86ee169f0329483a2b0f0d94bc20d81c11105"}
Jul 07 17:41:53.536442 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.536414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tnmvh" event={"ID":"1967f80f-c13c-46b2-91e5-067f3d56691c","Type":"ContainerStarted","Data":"33361af266c15eaefe5fbd6e56a831e7c17729faa48cffc7a81211a949857cd8"}
Jul 07 17:41:53.536606 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.536585    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:41:53.538122 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.538085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2s8zn" event={"ID":"60c44b63-0ee5-4f79-94c3-b2538bfb6088","Type":"ContainerStarted","Data":"81c3085de6fbfa246dd27f7a5a4dcc7d6ca9094d5b9ac924e6c28a522c2f4609"}
Jul 07 17:41:53.541145 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.541120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s6cr" event={"ID":"e6523d94-f705-49fb-ac51-83169974230c","Type":"ContainerStarted","Data":"2ceeb49b7b1ad917b332e53e6d082103b7bba5151a8c22e46b0e7ac597af90e4"}
Jul 07 17:41:53.541240 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.541170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s6cr" event={"ID":"e6523d94-f705-49fb-ac51-83169974230c","Type":"ContainerStarted","Data":"17edb761e9b694ececb34cc07de4afa9f83ffef9da6e60a1322fc031cd93f9c5"}
Jul 07 17:41:53.544665 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.543608    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56968984f8-2btr9" podStartSLOduration=2.118988268 podStartE2EDuration="10.543591109s" totalImagesPullingTime="8.424602841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.543139813 +0000 UTC m=+75.117087155" watchObservedRunningTime="2026-07-07 17:41:53.543591109 +0000 UTC m=+75.117538455"
Jul 07 17:41:53.563900 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.563851    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tnmvh" podStartSLOduration=67.048864145 podStartE2EDuration="1m14.563835805s" totalImagesPullingTime="7.51497166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.562817077 +0000 UTC m=+75.136764422" watchObservedRunningTime="2026-07-07 17:41:53.563835805 +0000 UTC m=+75.137783146"
Jul 07 17:41:53.585322 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.585268    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5878976f55-2r529" podStartSLOduration=2.18348362 podStartE2EDuration="10.58524896s" totalImagesPullingTime="8.40176534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.584620304 +0000 UTC m=+75.158567645" watchObservedRunningTime="2026-07-07 17:41:53.58524896 +0000 UTC m=+75.159196302"
Jul 07 17:41:53.632196 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.631716    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6s6cr" podStartSLOduration=67.133997034 podStartE2EDuration="1m14.631693828s" totalImagesPullingTime="7.497696794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.631459087 +0000 UTC m=+75.205406430" watchObservedRunningTime="2026-07-07 17:41:53.631693828 +0000 UTC m=+75.205641166"
Jul 07 17:41:53.632196 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.631917    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4rd2v" podStartSLOduration=67.063048793 podStartE2EDuration="1m14.631910607s" totalImagesPullingTime="7.568861814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:41:53.608485593 +0000 UTC m=+75.182432935" watchObservedRunningTime="2026-07-07 17:41:53.631910607 +0000 UTC m=+75.205857950"
Jul 07 17:41:53.817814 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:53.817733    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:41:57.557075 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:57.556992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2s8zn" event={"ID":"60c44b63-0ee5-4f79-94c3-b2538bfb6088","Type":"ContainerStarted","Data":"33cb84880e30cddfdb9a019e528bff76ff4f76d8f8b913e7415adacbcc618c17"}
Jul 07 17:41:57.557075 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:57.557046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2s8zn" event={"ID":"60c44b63-0ee5-4f79-94c3-b2538bfb6088","Type":"ContainerStarted","Data":"f77ebb31b4f494259974b25d4cc66efdcaba8c23a38a3b01911ba5cc53392bcb"}
Jul 07 17:41:59.847796 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.847763    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tkpch"]
Jul 07 17:41:59.852212 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.851798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.858466 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.858445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 17:41:59.858580 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.858531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l27bs\""
Jul 07 17:41:59.860813 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.860652    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 17:41:59.860813 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.860659    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 17:41:59.860813 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.860718    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 17:41:59.861045 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.860822    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 17:41:59.861215 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.861201    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 17:41:59.865751 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.865731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-sys\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.865824 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.865770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-tls\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.865824 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.865798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.865925 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.865826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-root\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.865981 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.865928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-textfile\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.866044 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.865990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4032b-4c92-460a-b54f-4c142e2244cc-metrics-client-ca\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.866044 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.866013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jw47\" (UniqueName: \"kubernetes.io/projected/b7d4032b-4c92-460a-b54f-4c142e2244cc-kube-api-access-8jw47\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.866173 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.866042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-wtmp\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.866173 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.866071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966505 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4032b-4c92-460a-b54f-4c142e2244cc-metrics-client-ca\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jw47\" (UniqueName: \"kubernetes.io/projected/b7d4032b-4c92-460a-b54f-4c142e2244cc-kube-api-access-8jw47\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-wtmp\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-sys\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-tls\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.966701 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-root\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967087 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.966748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-textfile\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967087 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.967058    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-textfile\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967087 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.967079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-sys\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967260 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.967195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d4032b-4c92-460a-b54f-4c142e2244cc-metrics-client-ca\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.967270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-wtmp\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.967273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d4032b-4c92-460a-b54f-4c142e2244cc-root\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.967510 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.967493    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.969221 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.969202    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.969411 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.969396    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d4032b-4c92-460a-b54f-4c142e2244cc-node-exporter-tls\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:41:59.986605 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:41:59.986569    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jw47\" (UniqueName: \"kubernetes.io/projected/b7d4032b-4c92-460a-b54f-4c142e2244cc-kube-api-access-8jw47\") pod \"node-exporter-tkpch\" (UID: \"b7d4032b-4c92-460a-b54f-4c142e2244cc\") " pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:42:00.167932 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:00.167782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tkpch"
Jul 07 17:42:00.179583 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:42:00.179557    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d4032b_4c92_460a_b54f_4c142e2244cc.slice/crio-ee390b23b58c3a96a89a2f95a86c5c97d876f340699903cb81acb02819780c1b WatchSource:0}: Error finding container ee390b23b58c3a96a89a2f95a86c5c97d876f340699903cb81acb02819780c1b: Status 404 returned error can't find the container with id ee390b23b58c3a96a89a2f95a86c5c97d876f340699903cb81acb02819780c1b
Jul 07 17:42:00.570700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:00.570668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkpch" event={"ID":"b7d4032b-4c92-460a-b54f-4c142e2244cc","Type":"ContainerStarted","Data":"ee390b23b58c3a96a89a2f95a86c5c97d876f340699903cb81acb02819780c1b"}
Jul 07 17:42:00.572642 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:00.572603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2s8zn" event={"ID":"60c44b63-0ee5-4f79-94c3-b2538bfb6088","Type":"ContainerStarted","Data":"6ab7e16fac29939d5989d2ab24bbce14371bf5bab303698b163b776379309e72"}
Jul 07 17:42:00.595280 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:00.595222    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2s8zn" podStartSLOduration=37.780242866 podStartE2EDuration="44.59520591s" totalImagesPullingTime="6.814963044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:41:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:42:00.594342274 +0000 UTC m=+82.168289620" watchObservedRunningTime="2026-07-07 17:42:00.59520591 +0000 UTC m=+82.169153269"
Jul 07 17:42:01.577682 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:01.577647    2577 generic.go:356] "Generic (PLEG): container finished" podID="b7d4032b-4c92-460a-b54f-4c142e2244cc" containerID="f6e0f835ac5aa7dca5878218b2627f315375354afe27d31807295f501060e3ba" exitCode=0
Jul 07 17:42:01.578083 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:01.577732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkpch" event={"ID":"b7d4032b-4c92-460a-b54f-4c142e2244cc","Type":"ContainerDied","Data":"f6e0f835ac5aa7dca5878218b2627f315375354afe27d31807295f501060e3ba"}
Jul 07 17:42:02.583755 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:02.583717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkpch" event={"ID":"b7d4032b-4c92-460a-b54f-4c142e2244cc","Type":"ContainerStarted","Data":"9f5400201b1e966ff66f47c6366c9cc5bb72ee18d88bd9c225849a617924bbe6"}
Jul 07 17:42:02.584148 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:02.583781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tkpch" event={"ID":"b7d4032b-4c92-460a-b54f-4c142e2244cc","Type":"ContainerStarted","Data":"ee7a6f2fa0a53627cf8b0506a98b33d4172f83bde6eeea64ce788ea7db739056"}
Jul 07 17:42:02.613129 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:02.613063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tkpch" podStartSLOduration=2.484467434 podStartE2EDuration="3.613041697s" totalImagesPullingTime="1.06571427s" totalInitContainerRuntime="62.859993ms" isStatefulPod=true podCreationTimestamp="2026-07-07 17:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:42:02.612191494 +0000 UTC m=+84.186138835" watchObservedRunningTime="2026-07-07 17:42:02.613041697 +0000 UTC m=+84.186989041"
Jul 07 17:42:06.503177 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:06.503149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79768b874c-tskt2"
Jul 07 17:42:08.830076 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:08.830036    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj" podUID="b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" containerName="registry" containerID="cri-o://bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab" gracePeriod=30
Jul 07 17:42:09.068941 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.068917    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:42:09.136979 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.136909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-ca-trust-extracted\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.136979 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.136939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjxg8\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-kube-api-access-xjxg8\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.136979 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.136972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-certificates\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.137250 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.136994    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-image-registry-private-configuration\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.137250 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-trusted-ca\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.137250 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.137250 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-installation-pull-secrets\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.137445 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-bound-sa-token\") pod \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\" (UID: \"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d\") "
Jul 07 17:42:09.137445 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137406    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:42:09.137542 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137521    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-certificates\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.137595 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.137550    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:42:09.139585 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.139557    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:42:09.139757 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.139636    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-kube-api-access-xjxg8" (OuterVolumeSpecName: "kube-api-access-xjxg8") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "kube-api-access-xjxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:42:09.139866 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.139758    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:42:09.139917 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.139868    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:42:09.139917 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.139889    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:42:09.145601 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.145575    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" (UID: "b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:42:09.237992 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.237946    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-trusted-ca\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.237992 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.237988    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-registry-tls\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.237992 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.237998    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-installation-pull-secrets\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.237992 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.238008    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-bound-sa-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.238271 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.238017    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-ca-trust-extracted\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.238271 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.238025    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxg8\" (UniqueName: \"kubernetes.io/projected/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-kube-api-access-xjxg8\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.238271 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.238034    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d-image-registry-private-configuration\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:42:09.609023 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.608986    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" containerID="bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab" exitCode=0
Jul 07 17:42:09.609223 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.609045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj" event={"ID":"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d","Type":"ContainerDied","Data":"bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab"}
Jul 07 17:42:09.609223 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.609055    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"
Jul 07 17:42:09.609223 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.609092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj" event={"ID":"b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d","Type":"ContainerDied","Data":"99a5a6119b24e0732df0dc54ef5396124bc4134b19b64208560dabf3e1c3da09"}
Jul 07 17:42:09.609223 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.609159    2577 scope.go:117] "RemoveContainer" containerID="bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab"
Jul 07 17:42:09.617998 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.617975    2577 scope.go:117] "RemoveContainer" containerID="bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab"
Jul 07 17:42:09.618275 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:42:09.618256    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab\": container with ID starting with bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab not found: ID does not exist" containerID="bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab"
Jul 07 17:42:09.618331 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.618284    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab"} err="failed to get container status \"bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab\": rpc error: code = NotFound desc = could not find container \"bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab\": container with ID starting with bdee62ce7dcf0c71a2083c2e4da2d56738f470da5812ba83f71c168f866f7cab not found: ID does not exist"
Jul 07 17:42:09.630828 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.630794    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"]
Jul 07 17:42:09.635198 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:09.635178    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6dfcf5f8d7-xr5vj"]
Jul 07 17:42:11.095465 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:11.095434    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" path="/var/lib/kubelet/pods/b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d/volumes"
Jul 07 17:42:24.547352 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:24.547319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tnmvh"
Jul 07 17:42:34.651250 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:34.651226    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkpch_b7d4032b-4c92-460a-b54f-4c142e2244cc/init-textfile/0.log"
Jul 07 17:42:34.851796 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:34.851767    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkpch_b7d4032b-4c92-460a-b54f-4c142e2244cc/node-exporter/0.log"
Jul 07 17:42:35.050514 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:35.050487    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkpch_b7d4032b-4c92-460a-b54f-4c142e2244cc/kube-rbac-proxy/0.log"
Jul 07 17:42:39.050685 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:42:39.050654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-gkbmq_a44a5ab6-5c81-4e25-8e12-4e78eaa1624a/networking-console-plugin/0.log"
Jul 07 17:45:38.858742 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:45:38.858712    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:45:38.861412 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:45:38.861391    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 17:45:38.863349 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:45:38.863331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:46:22.808371 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.808338    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"]
Jul 07 17:46:22.808795 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.808590    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" containerName="registry"
Jul 07 17:46:22.808795 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.808601    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" containerName="registry"
Jul 07 17:46:22.808795 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.808657    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ebfaf9-4924-4b1f-93e2-9ba8c98fb76d" containerName="registry"
Jul 07 17:46:22.811377 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.811361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:22.813843 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.813812    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 17:46:22.813982 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.813893    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 17:46:22.814632 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.814619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zqc76\""
Jul 07 17:46:22.821383 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.821361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"]
Jul 07 17:46:22.933637 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.933609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77c4489e-d8e3-460f-ae46-ab6f2e098fa6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qc9lm\" (UID: \"77c4489e-d8e3-460f-ae46-ab6f2e098fa6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:22.933791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:22.933672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt74x\" (UniqueName: \"kubernetes.io/projected/77c4489e-d8e3-460f-ae46-ab6f2e098fa6-kube-api-access-xt74x\") pod \"openshift-lws-operator-fd8ccff4c-qc9lm\" (UID: \"77c4489e-d8e3-460f-ae46-ab6f2e098fa6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:23.034173 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.034139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt74x\" (UniqueName: \"kubernetes.io/projected/77c4489e-d8e3-460f-ae46-ab6f2e098fa6-kube-api-access-xt74x\") pod \"openshift-lws-operator-fd8ccff4c-qc9lm\" (UID: \"77c4489e-d8e3-460f-ae46-ab6f2e098fa6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:23.034308 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.034193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77c4489e-d8e3-460f-ae46-ab6f2e098fa6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qc9lm\" (UID: \"77c4489e-d8e3-460f-ae46-ab6f2e098fa6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:23.034496 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.034479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77c4489e-d8e3-460f-ae46-ab6f2e098fa6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qc9lm\" (UID: \"77c4489e-d8e3-460f-ae46-ab6f2e098fa6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:23.045865 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.045838    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt74x\" (UniqueName: \"kubernetes.io/projected/77c4489e-d8e3-460f-ae46-ab6f2e098fa6-kube-api-access-xt74x\") pod \"openshift-lws-operator-fd8ccff4c-qc9lm\" (UID: \"77c4489e-d8e3-460f-ae46-ab6f2e098fa6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:23.119713 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.119643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"
Jul 07 17:46:23.243726 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.243693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm"]
Jul 07 17:46:23.246542 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:46:23.246512    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c4489e_d8e3_460f_ae46_ab6f2e098fa6.slice/crio-70858c57dd38e4063d2decc8855d950edaba97ae4022dfb94e4a02af41c15b4a WatchSource:0}: Error finding container 70858c57dd38e4063d2decc8855d950edaba97ae4022dfb94e4a02af41c15b4a: Status 404 returned error can't find the container with id 70858c57dd38e4063d2decc8855d950edaba97ae4022dfb94e4a02af41c15b4a
Jul 07 17:46:23.248049 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.248032    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:46:23.439897 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:23.439819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm" event={"ID":"77c4489e-d8e3-460f-ae46-ab6f2e098fa6","Type":"ContainerStarted","Data":"70858c57dd38e4063d2decc8855d950edaba97ae4022dfb94e4a02af41c15b4a"}
Jul 07 17:46:26.453126 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:26.453086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm" event={"ID":"77c4489e-d8e3-460f-ae46-ab6f2e098fa6","Type":"ContainerStarted","Data":"b721f107c9b59eed7f350d19f1c6a4d49542922304c63ee90716d760eb4b1d5f"}
Jul 07 17:46:26.474129 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:26.474067    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qc9lm" podStartSLOduration=1.744245305 podStartE2EDuration="4.474051891s" totalImagesPullingTime="2.729806586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:46:26.47349509 +0000 UTC m=+348.047442457" watchObservedRunningTime="2026-07-07 17:46:26.474051891 +0000 UTC m=+348.047999231"
Jul 07 17:46:46.177874 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.177839    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"]
Jul 07 17:46:46.182099 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.182077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.185400 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.185383    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 17:46:46.185779 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.185760    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-727k9\""
Jul 07 17:46:46.185828 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.185780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 17:46:46.186153 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.186136    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 17:46:46.186237 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.186208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 17:46:46.236864 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.236830    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"]
Jul 07 17:46:46.317212 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.317176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.317396 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.317229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ngz\" (UniqueName: \"kubernetes.io/projected/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-kube-api-access-q6ngz\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.317396 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.317270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-tmp\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.317396 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.317292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.317396 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.317342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-manifests\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.418597 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.418564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-manifests\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.418784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.418615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.418784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.418665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ngz\" (UniqueName: \"kubernetes.io/projected/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-kube-api-access-q6ngz\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.418784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.418701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-tmp\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.418784 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.418726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.419062 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.419038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-manifests\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.421005 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.420987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-tmp\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.421310 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.421288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.421356 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.421313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.428839 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.428789    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ngz\" (UniqueName: \"kubernetes.io/projected/032c7bb8-1e8a-4ec3-a667-cd81e01053d8-kube-api-access-q6ngz\") pod \"opendatahub-operator-controller-manager-6f77d785b9-5hwvh\" (UID: \"032c7bb8-1e8a-4ec3-a667-cd81e01053d8\") " pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.491811 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.491779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:46.636516 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:46.636482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"]
Jul 07 17:46:46.641719 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:46:46.641692    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032c7bb8_1e8a_4ec3_a667_cd81e01053d8.slice/crio-0668e46c2036b512b5d27872bfdaac6136f21ffc8aa23c369652a9925ba659a6 WatchSource:0}: Error finding container 0668e46c2036b512b5d27872bfdaac6136f21ffc8aa23c369652a9925ba659a6: Status 404 returned error can't find the container with id 0668e46c2036b512b5d27872bfdaac6136f21ffc8aa23c369652a9925ba659a6
Jul 07 17:46:47.533286 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:47.533243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh" event={"ID":"032c7bb8-1e8a-4ec3-a667-cd81e01053d8","Type":"ContainerStarted","Data":"0668e46c2036b512b5d27872bfdaac6136f21ffc8aa23c369652a9925ba659a6"}
Jul 07 17:46:49.541754 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:49.541721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh" event={"ID":"032c7bb8-1e8a-4ec3-a667-cd81e01053d8","Type":"ContainerStarted","Data":"ce97f012ebba05ed5c2e30a4b0cc88f1b573b563c1facebf84270ac9123233ff"}
Jul 07 17:46:51.550677 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:51.550640    2577 generic.go:356] "Generic (PLEG): container finished" podID="032c7bb8-1e8a-4ec3-a667-cd81e01053d8" containerID="ce97f012ebba05ed5c2e30a4b0cc88f1b573b563c1facebf84270ac9123233ff" exitCode=0
Jul 07 17:46:51.551067 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:51.550716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh" event={"ID":"032c7bb8-1e8a-4ec3-a667-cd81e01053d8","Type":"ContainerDied","Data":"ce97f012ebba05ed5c2e30a4b0cc88f1b573b563c1facebf84270ac9123233ff"}
Jul 07 17:46:52.556324 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:52.556286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh" event={"ID":"032c7bb8-1e8a-4ec3-a667-cd81e01053d8","Type":"ContainerStarted","Data":"5f37ba24ce24c4dd4981dcf845386738540d0bad44bc5c6c073422d652c71b25"}
Jul 07 17:46:52.556725 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:52.556533    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:46:52.585918 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:46:52.585861    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh" podStartSLOduration=2.3851622199999998 podStartE2EDuration="6.585838426s" totalImagesPullingTime="3.172965563s" totalInitContainerRuntime="1.027710643s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:46:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:46:52.584467666 +0000 UTC m=+374.158415008" watchObservedRunningTime="2026-07-07 17:46:52.585838426 +0000 UTC m=+374.159785772"
Jul 07 17:47:03.562290 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:03.562253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f77d785b9-5hwvh"
Jul 07 17:47:06.627348 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.627313    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"]
Jul 07 17:47:06.630526 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.630508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.633819 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.633788    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 17:47:06.633979 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.633961    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 17:47:06.634754 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.634732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 17:47:06.634813 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.634737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-99jl5\""
Jul 07 17:47:06.634878 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.634865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 17:47:06.655290 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.655262    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"]
Jul 07 17:47:06.664431 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.664397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6jk\" (UniqueName: \"kubernetes.io/projected/51fba43c-8cfe-4daf-bf0c-9841ca500aec-kube-api-access-fx6jk\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.664575 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.664449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fba43c-8cfe-4daf-bf0c-9841ca500aec-tls-certs\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.664575 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.664474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fba43c-8cfe-4daf-bf0c-9841ca500aec-tmp\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.765542 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.765507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6jk\" (UniqueName: \"kubernetes.io/projected/51fba43c-8cfe-4daf-bf0c-9841ca500aec-kube-api-access-fx6jk\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.765719 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.765560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fba43c-8cfe-4daf-bf0c-9841ca500aec-tls-certs\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.765719 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.765585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fba43c-8cfe-4daf-bf0c-9841ca500aec-tmp\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.767790 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.767758    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fba43c-8cfe-4daf-bf0c-9841ca500aec-tmp\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.767970 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.767951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fba43c-8cfe-4daf-bf0c-9841ca500aec-tls-certs\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.802842 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.802807    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6jk\" (UniqueName: \"kubernetes.io/projected/51fba43c-8cfe-4daf-bf0c-9841ca500aec-kube-api-access-fx6jk\") pod \"kube-auth-proxy-5675bf5659-c5zcw\" (UID: \"51fba43c-8cfe-4daf-bf0c-9841ca500aec\") " pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:06.939166 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:06.939084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"
Jul 07 17:47:07.001065 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.000995    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lbzc6"]
Jul 07 17:47:07.004339 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.004315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.007949 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.007924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 07 17:47:07.008332 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.008318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dw6zq\""
Jul 07 17:47:07.050175 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.050145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lbzc6"]
Jul 07 17:47:07.068226 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.068192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq42q\" (UniqueName: \"kubernetes.io/projected/4e99d36f-1252-4e68-9eab-2d6a40e88192-kube-api-access-zq42q\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.068383 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.068245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.169514 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.169484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.169773 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:07.169664    2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 17:47:07.169886 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.169803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq42q\" (UniqueName: \"kubernetes.io/projected/4e99d36f-1252-4e68-9eab-2d6a40e88192-kube-api-access-zq42q\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.169886 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:07.169822    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert podName:4e99d36f-1252-4e68-9eab-2d6a40e88192 nodeName:}" failed. No retries permitted until 2026-07-07 17:47:07.669793375 +0000 UTC m=+389.243740700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert") pod "odh-model-controller-7998c67dc4-lbzc6" (UID: "4e99d36f-1252-4e68-9eab-2d6a40e88192") : secret "odh-model-controller-webhook-cert" not found
Jul 07 17:47:07.196411 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.196329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw"]
Jul 07 17:47:07.196970 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.196949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq42q\" (UniqueName: \"kubernetes.io/projected/4e99d36f-1252-4e68-9eab-2d6a40e88192-kube-api-access-zq42q\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.201020 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:07.200994    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fba43c_8cfe_4daf_bf0c_9841ca500aec.slice/crio-00f264cc8779d76076c0f13edfe82a7a13f2d4c68f89ad1bdec4ac38d2435650 WatchSource:0}: Error finding container 00f264cc8779d76076c0f13edfe82a7a13f2d4c68f89ad1bdec4ac38d2435650: Status 404 returned error can't find the container with id 00f264cc8779d76076c0f13edfe82a7a13f2d4c68f89ad1bdec4ac38d2435650
Jul 07 17:47:07.609695 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.609657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw" event={"ID":"51fba43c-8cfe-4daf-bf0c-9841ca500aec","Type":"ContainerStarted","Data":"00f264cc8779d76076c0f13edfe82a7a13f2d4c68f89ad1bdec4ac38d2435650"}
Jul 07 17:47:07.672580 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:07.672543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:07.672998 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:07.672662    2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 17:47:07.672998 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:07.672734    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert podName:4e99d36f-1252-4e68-9eab-2d6a40e88192 nodeName:}" failed. No retries permitted until 2026-07-07 17:47:08.672716403 +0000 UTC m=+390.246663727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert") pod "odh-model-controller-7998c67dc4-lbzc6" (UID: "4e99d36f-1252-4e68-9eab-2d6a40e88192") : secret "odh-model-controller-webhook-cert" not found
Jul 07 17:47:08.522795 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.522756    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pzgtv"]
Jul 07 17:47:08.526033 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.526012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:08.529663 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.529640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 07 17:47:08.578317 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.578283    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pzgtv"]
Jul 07 17:47:08.580029 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.579995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2pc\" (UniqueName: \"kubernetes.io/projected/f88f0257-449e-4759-b23d-f9f1d36acce7-kube-api-access-jf2pc\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:08.580029 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.580030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88f0257-449e-4759-b23d-f9f1d36acce7-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:08.681455 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.681416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:08.681949 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.681466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2pc\" (UniqueName: \"kubernetes.io/projected/f88f0257-449e-4759-b23d-f9f1d36acce7-kube-api-access-jf2pc\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:08.681949 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.681496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88f0257-449e-4759-b23d-f9f1d36acce7-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:08.681949 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:08.681604    2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 17:47:08.681949 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:08.681696    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f88f0257-449e-4759-b23d-f9f1d36acce7-tls-certs podName:f88f0257-449e-4759-b23d-f9f1d36acce7 nodeName:}" failed. No retries permitted until 2026-07-07 17:47:09.181676394 +0000 UTC m=+390.755623732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f88f0257-449e-4759-b23d-f9f1d36acce7-tls-certs") pod "model-serving-api-54d89bd5fc-pzgtv" (UID: "f88f0257-449e-4759-b23d-f9f1d36acce7") : secret "model-serving-api-tls" not found
Jul 07 17:47:08.683987 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.683962    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e99d36f-1252-4e68-9eab-2d6a40e88192-cert\") pod \"odh-model-controller-7998c67dc4-lbzc6\" (UID: \"4e99d36f-1252-4e68-9eab-2d6a40e88192\") " pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:08.697294 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.697270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2pc\" (UniqueName: \"kubernetes.io/projected/f88f0257-449e-4759-b23d-f9f1d36acce7-kube-api-access-jf2pc\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:08.816444 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.816346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:08.960572 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:08.960535    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lbzc6"]
Jul 07 17:47:08.964349 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:08.964314    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e99d36f_1252_4e68_9eab_2d6a40e88192.slice/crio-b226ca11deabff55277abd857e1e97589040721f45cedd3754d448b52ef0512b WatchSource:0}: Error finding container b226ca11deabff55277abd857e1e97589040721f45cedd3754d448b52ef0512b: Status 404 returned error can't find the container with id b226ca11deabff55277abd857e1e97589040721f45cedd3754d448b52ef0512b
Jul 07 17:47:09.185093 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.185004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88f0257-449e-4759-b23d-f9f1d36acce7-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:09.187332 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.187313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f88f0257-449e-4759-b23d-f9f1d36acce7-tls-certs\") pod \"model-serving-api-54d89bd5fc-pzgtv\" (UID: \"f88f0257-449e-4759-b23d-f9f1d36acce7\") " pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:09.395637 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.395604    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"]
Jul 07 17:47:09.399948 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.399927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.405248 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.405016    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 07 17:47:09.405248 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.405040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 17:47:09.405248 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.405065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 07 17:47:09.405248 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.405169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k6xgg\""
Jul 07 17:47:09.405526 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.405378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 07 17:47:09.431446 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.431419    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"]
Jul 07 17:47:09.439723 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.439651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:09.486957 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.486783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.487603 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.487541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd372e5b-4d38-497b-b499-7bf8523b66ab-local-certs\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.487749 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.487616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-cacerts\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.487749 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.487642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.487749 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.487692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.487749 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.487718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-token\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.487927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.487749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6g6\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-kube-api-access-px6g6\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.586491 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.586403    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-pzgtv"]
Jul 07 17:47:09.588894 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.588865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589020 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:09.588896    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f0257_449e_4759_b23d_f9f1d36acce7.slice/crio-7039db137104cbfb24e20324d1029cb3f20ddc90755ee151a45dd8480353f27d WatchSource:0}: Error finding container 7039db137104cbfb24e20324d1029cb3f20ddc90755ee151a45dd8480353f27d: Status 404 returned error can't find the container with id 7039db137104cbfb24e20324d1029cb3f20ddc90755ee151a45dd8480353f27d
Jul 07 17:47:09.589020 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.588926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd372e5b-4d38-497b-b499-7bf8523b66ab-local-certs\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589020 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.588954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-cacerts\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589020 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.588980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589280 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.589026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589280 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.589052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-token\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589280 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.589075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6g6\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-kube-api-access-px6g6\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.589887 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.589835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.591600 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.591507    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-cacerts\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.591679 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.591612    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd372e5b-4d38-497b-b499-7bf8523b66ab-local-certs\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.591735 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.591701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.591803 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.591787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.602206 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.602179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6g6\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-kube-api-access-px6g6\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.604012 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.603991    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-token\") pod \"istiod-openshift-gateway-7574459656-ssgdv\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.618674 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.618646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv" event={"ID":"f88f0257-449e-4759-b23d-f9f1d36acce7","Type":"ContainerStarted","Data":"7039db137104cbfb24e20324d1029cb3f20ddc90755ee151a45dd8480353f27d"}
Jul 07 17:47:09.619889 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.619857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6" event={"ID":"4e99d36f-1252-4e68-9eab-2d6a40e88192","Type":"ContainerStarted","Data":"b226ca11deabff55277abd857e1e97589040721f45cedd3754d448b52ef0512b"}
Jul 07 17:47:09.711609 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.711492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:09.896349 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:09.896323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"]
Jul 07 17:47:10.559908 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:10.559869    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd372e5b_4d38_497b_b499_7bf8523b66ab.slice/crio-484ed2ee358e7c11bc3e310495f4cbc5d54167797b6783e98fc0087af7f91a24 WatchSource:0}: Error finding container 484ed2ee358e7c11bc3e310495f4cbc5d54167797b6783e98fc0087af7f91a24: Status 404 returned error can't find the container with id 484ed2ee358e7c11bc3e310495f4cbc5d54167797b6783e98fc0087af7f91a24
Jul 07 17:47:10.626039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:10.626000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv" event={"ID":"fd372e5b-4d38-497b-b499-7bf8523b66ab","Type":"ContainerStarted","Data":"484ed2ee358e7c11bc3e310495f4cbc5d54167797b6783e98fc0087af7f91a24"}
Jul 07 17:47:12.638233 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.638149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv" event={"ID":"f88f0257-449e-4759-b23d-f9f1d36acce7","Type":"ContainerStarted","Data":"77ba26f8704af76c3e6a818b8c62345c03d9d678932c2ae088d49f0dc86fdb7c"}
Jul 07 17:47:12.638685 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.638246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:12.640064 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.640014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw" event={"ID":"51fba43c-8cfe-4daf-bf0c-9841ca500aec","Type":"ContainerStarted","Data":"91ae1b664c021a415946005ca11b294b623dded9bfbae426ff671c52de643d32"}
Jul 07 17:47:12.641909 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.641883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6" event={"ID":"4e99d36f-1252-4e68-9eab-2d6a40e88192","Type":"ContainerStarted","Data":"874cc6f048abb911f5ddfb12045324191a6664adc7a456f9b7e60a7164f0960f"}
Jul 07 17:47:12.642049 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.641948    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:12.660571 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.660518    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv" podStartSLOduration=1.93948638 podStartE2EDuration="4.660501129s" totalImagesPullingTime="2.721014749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:12.659349194 +0000 UTC m=+394.233296539" watchObservedRunningTime="2026-07-07 17:47:12.660501129 +0000 UTC m=+394.234448471"
Jul 07 17:47:12.679228 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.678995    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6" podStartSLOduration=3.32693599 podStartE2EDuration="6.678975899s" totalImagesPullingTime="3.352039909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:12.676901264 +0000 UTC m=+394.250848633" watchObservedRunningTime="2026-07-07 17:47:12.678975899 +0000 UTC m=+394.252923241"
Jul 07 17:47:12.710502 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:12.710445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5675bf5659-c5zcw" podStartSLOduration=1.660804723 podStartE2EDuration="6.710426336s" totalImagesPullingTime="5.049621613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:12.709748563 +0000 UTC m=+394.283695905" watchObservedRunningTime="2026-07-07 17:47:12.710426336 +0000 UTC m=+394.284373705"
Jul 07 17:47:13.834490 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:13.834450    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:47:13.834797 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:13.834515    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:47:14.652088 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:14.652053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv" event={"ID":"fd372e5b-4d38-497b-b499-7bf8523b66ab","Type":"ContainerStarted","Data":"bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634"}
Jul 07 17:47:14.692540 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:14.692463    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv" podStartSLOduration=2.420446933 podStartE2EDuration="5.692444049s" totalImagesPullingTime="3.271997116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:14.691437882 +0000 UTC m=+396.265385268" watchObservedRunningTime="2026-07-07 17:47:14.692444049 +0000 UTC m=+396.266391391"
Jul 07 17:47:15.090033 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.089996    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"]
Jul 07 17:47:15.093613 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.093586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.096301 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.096281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-v9ckn\""
Jul 07 17:47:15.115177 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.115151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"]
Jul 07 17:47:15.138718 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.138858 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.138858 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.138858 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.138858 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.138858 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9nz\" (UniqueName: \"kubernetes.io/projected/e081b8b6-ee75-44a4-b5b3-796ca0309b99-kube-api-access-jf9nz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.139041 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.139041 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.139041 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.138936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239516 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239516 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239758 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9nz\" (UniqueName: \"kubernetes.io/projected/e081b8b6-ee75-44a4-b5b3-796ca0309b99-kube-api-access-jf9nz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239758 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239758 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239904 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239904 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239904 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.239904 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.239884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.240282 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.240251    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.240429 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.240344    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.240429 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.240406    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.240548 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.240444    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.240548 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.240453    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.241955 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.241927    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.242185 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.242167    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.254524 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.254496    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e081b8b6-ee75-44a4-b5b3-796ca0309b99-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.254524 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.254516    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9nz\" (UniqueName: \"kubernetes.io/projected/e081b8b6-ee75-44a4-b5b3-796ca0309b99-kube-api-access-jf9nz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rq928\" (UID: \"e081b8b6-ee75-44a4-b5b3-796ca0309b99\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.403416 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.403327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:15.565277 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.565252    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"]
Jul 07 17:47:15.566132 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:15.566090    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode081b8b6_ee75_44a4_b5b3_796ca0309b99.slice/crio-62e66ef97fddfec72a49dedb83636c09cb33edfce821295f48d86f2bbcd6b0c9 WatchSource:0}: Error finding container 62e66ef97fddfec72a49dedb83636c09cb33edfce821295f48d86f2bbcd6b0c9: Status 404 returned error can't find the container with id 62e66ef97fddfec72a49dedb83636c09cb33edfce821295f48d86f2bbcd6b0c9
Jul 07 17:47:15.657735 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.657648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928" event={"ID":"e081b8b6-ee75-44a4-b5b3-796ca0309b99","Type":"ContainerStarted","Data":"62e66ef97fddfec72a49dedb83636c09cb33edfce821295f48d86f2bbcd6b0c9"}
Jul 07 17:47:15.657876 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.657852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:15.659594 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:15.659572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:17.876208 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.876173    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"]
Jul 07 17:47:17.884178 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.884102    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"]
Jul 07 17:47:17.884323 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.884204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:17.887301 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.887244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 07 17:47:17.888333 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.888307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lwm8j\""
Jul 07 17:47:17.889029 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.889010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:17.892588 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.892030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wqz2w\""
Jul 07 17:47:17.892588 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.892065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 07 17:47:17.908433 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.908411    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"]
Jul 07 17:47:17.908630 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.908616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"]
Jul 07 17:47:17.967970 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.967932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cw76\" (UniqueName: \"kubernetes.io/projected/2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9-kube-api-access-5cw76\") pod \"kserve-controller-manager-d6f4b65b8-5cgnt\" (UID: \"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:17.968165 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.967990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxdl\" (UniqueName: \"kubernetes.io/projected/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-kube-api-access-9xxdl\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:17.968165 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.968076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:17.968165 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:17.968130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cgnt\" (UID: \"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:18.069453 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.069423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:18.069603 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.069460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cgnt\" (UID: \"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:18.069603 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.069542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cw76\" (UniqueName: \"kubernetes.io/projected/2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9-kube-api-access-5cw76\") pod \"kserve-controller-manager-d6f4b65b8-5cgnt\" (UID: \"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:18.069603 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:18.069581    2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 07 17:47:18.069746 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.069584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxdl\" (UniqueName: \"kubernetes.io/projected/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-kube-api-access-9xxdl\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:18.069746 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:18.069652    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-cert podName:84c02517-5eb7-4070-bd06-a9f0cd5f99ce nodeName:}" failed. No retries permitted until 2026-07-07 17:47:18.569629715 +0000 UTC m=+400.143577047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-cert") pod "llmisvc-controller-manager-777c6bd7d5-zlfbr" (UID: "84c02517-5eb7-4070-bd06-a9f0cd5f99ce") : secret "llmisvc-webhook-server-cert" not found
Jul 07 17:47:18.072173 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.072145    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9-cert\") pod \"kserve-controller-manager-d6f4b65b8-5cgnt\" (UID: \"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:18.104519 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.104488    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cw76\" (UniqueName: \"kubernetes.io/projected/2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9-kube-api-access-5cw76\") pod \"kserve-controller-manager-d6f4b65b8-5cgnt\" (UID: \"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:18.104637 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.104595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxdl\" (UniqueName: \"kubernetes.io/projected/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-kube-api-access-9xxdl\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:18.123001 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.122965    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:47:18.123127 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.123056    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:47:18.123127 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.123098    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:47:18.212622 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.212418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:18.389980 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.389953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"]
Jul 07 17:47:18.392266 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:18.392229    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8bbc4f_0ffd_434f_a887_b0f9dc3ad7a9.slice/crio-2f08877188b36a3401752749ebdc44bd2a59d06328e7322b2a663e0014df6008 WatchSource:0}: Error finding container 2f08877188b36a3401752749ebdc44bd2a59d06328e7322b2a663e0014df6008: Status 404 returned error can't find the container with id 2f08877188b36a3401752749ebdc44bd2a59d06328e7322b2a663e0014df6008
Jul 07 17:47:18.574816 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.574786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:18.577205 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.577179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c02517-5eb7-4070-bd06-a9f0cd5f99ce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zlfbr\" (UID: \"84c02517-5eb7-4070-bd06-a9f0cd5f99ce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:18.673896 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.673816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt" event={"ID":"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9","Type":"ContainerStarted","Data":"2f08877188b36a3401752749ebdc44bd2a59d06328e7322b2a663e0014df6008"}
Jul 07 17:47:18.675392 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.675363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928" event={"ID":"e081b8b6-ee75-44a4-b5b3-796ca0309b99","Type":"ContainerStarted","Data":"86ed88f9ed3d7915782c47bd7edccdc96d232a876b48a4a5deee5d83fe7df121"}
Jul 07 17:47:18.708458 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.707790    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928" podStartSLOduration=1.153233451 podStartE2EDuration="3.707768992s" totalImagesPullingTime="2.554535541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:18.706288166 +0000 UTC m=+400.280235506" watchObservedRunningTime="2026-07-07 17:47:18.707768992 +0000 UTC m=+400.281716333"
Jul 07 17:47:18.799007 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.798973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:18.949658 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:18.949585    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"]
Jul 07 17:47:18.964155 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:47:18.964124    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84c02517_5eb7_4070_bd06_a9f0cd5f99ce.slice/crio-c48886e6da3a54de26e9be4d7985fdd9dc561a7425dbc1d047219e7fce5d5c17 WatchSource:0}: Error finding container c48886e6da3a54de26e9be4d7985fdd9dc561a7425dbc1d047219e7fce5d5c17: Status 404 returned error can't find the container with id c48886e6da3a54de26e9be4d7985fdd9dc561a7425dbc1d047219e7fce5d5c17
Jul 07 17:47:19.007822 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.007784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"]
Jul 07 17:47:19.008023 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.007999    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv" podUID="fd372e5b-4d38-497b-b499-7bf8523b66ab" containerName="discovery" containerID="cri-o://bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634" gracePeriod=30
Jul 07 17:47:19.263034 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.263002    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:19.381902 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.381870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd372e5b-4d38-497b-b499-7bf8523b66ab-local-certs\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382089 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.381929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px6g6\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-kube-api-access-px6g6\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382089 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.381985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-cacerts\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382089 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.382024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-kubeconfig\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382089 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.382051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-ca-configmap\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382311 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.382143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-dns-cert\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382311 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.382181    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-token\") pod \"fd372e5b-4d38-497b-b499-7bf8523b66ab\" (UID: \"fd372e5b-4d38-497b-b499-7bf8523b66ab\") "
Jul 07 17:47:19.382644 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.382572    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 17:47:19.385022 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.384889    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:47:19.385022 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.384926    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-cacerts" (OuterVolumeSpecName: "cacerts") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:47:19.385022 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.384934    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd372e5b-4d38-497b-b499-7bf8523b66ab-local-certs" (OuterVolumeSpecName: "local-certs") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:47:19.385022 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.384987    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-kube-api-access-px6g6" (OuterVolumeSpecName: "kube-api-access-px6g6") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "kube-api-access-px6g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:47:19.385273 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.385065    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:47:19.385273 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.385162    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd372e5b-4d38-497b-b499-7bf8523b66ab" (UID: "fd372e5b-4d38-497b-b499-7bf8523b66ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:47:19.404333 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.404310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:19.409261 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.409239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:19.483054 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483022    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-dns-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.483054 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483051    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.483054 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483060    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd372e5b-4d38-497b-b499-7bf8523b66ab-local-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.483341 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483068    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6g6\" (UniqueName: \"kubernetes.io/projected/fd372e5b-4d38-497b-b499-7bf8523b66ab-kube-api-access-px6g6\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.483341 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483078    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-cacerts\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.483341 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483087    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-kubeconfig\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.483341 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.483095    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fd372e5b-4d38-497b-b499-7bf8523b66ab-istio-csr-ca-configmap\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:47:19.681267 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.681230    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd372e5b-4d38-497b-b499-7bf8523b66ab" containerID="bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634" exitCode=0
Jul 07 17:47:19.681456 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.681298    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"
Jul 07 17:47:19.681456 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.681308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv" event={"ID":"fd372e5b-4d38-497b-b499-7bf8523b66ab","Type":"ContainerDied","Data":"bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634"}
Jul 07 17:47:19.681456 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.681383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv" event={"ID":"fd372e5b-4d38-497b-b499-7bf8523b66ab","Type":"ContainerDied","Data":"484ed2ee358e7c11bc3e310495f4cbc5d54167797b6783e98fc0087af7f91a24"}
Jul 07 17:47:19.681456 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.681411    2577 scope.go:117] "RemoveContainer" containerID="bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634"
Jul 07 17:47:19.682599 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.682574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr" event={"ID":"84c02517-5eb7-4070-bd06-a9f0cd5f99ce","Type":"ContainerStarted","Data":"c48886e6da3a54de26e9be4d7985fdd9dc561a7425dbc1d047219e7fce5d5c17"}
Jul 07 17:47:19.682895 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.682872    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:19.683864 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.683845    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rq928"
Jul 07 17:47:19.690896 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.690881    2577 scope.go:117] "RemoveContainer" containerID="bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634"
Jul 07 17:47:19.691462 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:47:19.691423    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634\": container with ID starting with bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634 not found: ID does not exist" containerID="bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634"
Jul 07 17:47:19.691565 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.691453    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634"} err="failed to get container status \"bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634\": rpc error: code = NotFound desc = could not find container \"bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634\": container with ID starting with bb479830216e750f5842c4bc21a73fbb9a533181e591a2781e667efbf1662634 not found: ID does not exist"
Jul 07 17:47:19.768607 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.768577    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"]
Jul 07 17:47:19.781899 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:19.781863    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ssgdv"]
Jul 07 17:47:21.097920 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:21.097890    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd372e5b-4d38-497b-b499-7bf8523b66ab" path="/var/lib/kubelet/pods/fd372e5b-4d38-497b-b499-7bf8523b66ab/volumes"
Jul 07 17:47:22.698372 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:22.698341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt" event={"ID":"2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9","Type":"ContainerStarted","Data":"08e1df1e7719bb718e9252a83ba18862b82af377bc976b67b6adad5c552fb22a"}
Jul 07 17:47:22.698806 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:22.698446    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:47:22.699975 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:22.699953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr" event={"ID":"84c02517-5eb7-4070-bd06-a9f0cd5f99ce","Type":"ContainerStarted","Data":"c21400472e6a72b25d2b92ddf51ba9f361cb4b529b6b7591dbf438d89ef73e00"}
Jul 07 17:47:22.700119 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:22.700093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:22.717320 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:22.717274    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt" podStartSLOduration=1.9284923269999998 podStartE2EDuration="5.717259667s" totalImagesPullingTime="3.78876734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:22.715634062 +0000 UTC m=+404.289581403" watchObservedRunningTime="2026-07-07 17:47:22.717259667 +0000 UTC m=+404.291207030"
Jul 07 17:47:22.738522 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:22.738480    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr" podStartSLOduration=2.517526226 podStartE2EDuration="5.738464429s" totalImagesPullingTime="3.220938203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:47:22.73747824 +0000 UTC m=+404.311425580" watchObservedRunningTime="2026-07-07 17:47:22.738464429 +0000 UTC m=+404.312411834"
Jul 07 17:47:23.649433 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:23.649401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lbzc6"
Jul 07 17:47:23.651352 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:23.651332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-pzgtv"
Jul 07 17:47:53.712974 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:53.712944    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zlfbr"
Jul 07 17:47:53.716048 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:47:53.716028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5cgnt"
Jul 07 17:48:07.391076 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.390997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zw265"]
Jul 07 17:48:07.391435 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.391296    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd372e5b-4d38-497b-b499-7bf8523b66ab" containerName="discovery"
Jul 07 17:48:07.391435 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.391311    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd372e5b-4d38-497b-b499-7bf8523b66ab" containerName="discovery"
Jul 07 17:48:07.391435 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.391376    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd372e5b-4d38-497b-b499-7bf8523b66ab" containerName="discovery"
Jul 07 17:48:07.394008 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.393993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:07.403170 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.403149    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 17:48:07.403264 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.403156    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-c2c66\""
Jul 07 17:48:07.403264 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.403248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 17:48:07.428293 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.428266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zw265"]
Jul 07 17:48:07.484684 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.484645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmnm\" (UniqueName: \"kubernetes.io/projected/22f36e0f-3039-4900-83eb-0b4aa7bc395c-kube-api-access-7rmnm\") pod \"kuadrant-operator-catalog-zw265\" (UID: \"22f36e0f-3039-4900-83eb-0b4aa7bc395c\") " pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:07.585564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.585532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmnm\" (UniqueName: \"kubernetes.io/projected/22f36e0f-3039-4900-83eb-0b4aa7bc395c-kube-api-access-7rmnm\") pod \"kuadrant-operator-catalog-zw265\" (UID: \"22f36e0f-3039-4900-83eb-0b4aa7bc395c\") " pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:07.599715 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.599685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmnm\" (UniqueName: \"kubernetes.io/projected/22f36e0f-3039-4900-83eb-0b4aa7bc395c-kube-api-access-7rmnm\") pod \"kuadrant-operator-catalog-zw265\" (UID: \"22f36e0f-3039-4900-83eb-0b4aa7bc395c\") " pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:07.703272 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.703189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:07.715366 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.715337    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zw265"]
Jul 07 17:48:07.831926 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.831903    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zw265"]
Jul 07 17:48:07.834315 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:48:07.834284    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f36e0f_3039_4900_83eb_0b4aa7bc395c.slice/crio-0990e96301371fde6ed0a90dc445ebaae47f9a566c086ff2341542129ee7d295 WatchSource:0}: Error finding container 0990e96301371fde6ed0a90dc445ebaae47f9a566c086ff2341542129ee7d295: Status 404 returned error can't find the container with id 0990e96301371fde6ed0a90dc445ebaae47f9a566c086ff2341542129ee7d295
Jul 07 17:48:07.884500 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.884458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zw265" event={"ID":"22f36e0f-3039-4900-83eb-0b4aa7bc395c","Type":"ContainerStarted","Data":"0990e96301371fde6ed0a90dc445ebaae47f9a566c086ff2341542129ee7d295"}
Jul 07 17:48:07.928017 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.927985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q2mk9"]
Jul 07 17:48:07.932146 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.932130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:07.944929 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.944904    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q2mk9"]
Jul 07 17:48:07.988537 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:07.988460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7v55\" (UniqueName: \"kubernetes.io/projected/c8622673-b8f5-4cc6-bf73-a2139b185195-kube-api-access-k7v55\") pod \"kuadrant-operator-catalog-q2mk9\" (UID: \"c8622673-b8f5-4cc6-bf73-a2139b185195\") " pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:08.089826 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:08.089791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7v55\" (UniqueName: \"kubernetes.io/projected/c8622673-b8f5-4cc6-bf73-a2139b185195-kube-api-access-k7v55\") pod \"kuadrant-operator-catalog-q2mk9\" (UID: \"c8622673-b8f5-4cc6-bf73-a2139b185195\") " pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:08.099201 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:08.099179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7v55\" (UniqueName: \"kubernetes.io/projected/c8622673-b8f5-4cc6-bf73-a2139b185195-kube-api-access-k7v55\") pod \"kuadrant-operator-catalog-q2mk9\" (UID: \"c8622673-b8f5-4cc6-bf73-a2139b185195\") " pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:08.241299 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:08.241227    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:08.382994 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:08.382954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q2mk9"]
Jul 07 17:48:08.385025 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:48:08.385001    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8622673_b8f5_4cc6_bf73_a2139b185195.slice/crio-5bc34d6fcc5220515f1478e97a2ecb4f4860f36b60b1c80ea95b361e4fd6a6d5 WatchSource:0}: Error finding container 5bc34d6fcc5220515f1478e97a2ecb4f4860f36b60b1c80ea95b361e4fd6a6d5: Status 404 returned error can't find the container with id 5bc34d6fcc5220515f1478e97a2ecb4f4860f36b60b1c80ea95b361e4fd6a6d5
Jul 07 17:48:08.891153 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:08.891101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9" event={"ID":"c8622673-b8f5-4cc6-bf73-a2139b185195","Type":"ContainerStarted","Data":"5bc34d6fcc5220515f1478e97a2ecb4f4860f36b60b1c80ea95b361e4fd6a6d5"}
Jul 07 17:48:10.900270 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:10.900230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zw265" event={"ID":"22f36e0f-3039-4900-83eb-0b4aa7bc395c","Type":"ContainerStarted","Data":"c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700"}
Jul 07 17:48:10.900742 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:10.900312    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zw265" podUID="22f36e0f-3039-4900-83eb-0b4aa7bc395c" containerName="registry-server" containerID="cri-o://c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700" gracePeriod=2
Jul 07 17:48:10.901804 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:10.901781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9" event={"ID":"c8622673-b8f5-4cc6-bf73-a2139b185195","Type":"ContainerStarted","Data":"4e0c598f12bd2380ba8f44c5cc6a0d55034612331685453cea1b0fdd09d29ba9"}
Jul 07 17:48:10.916120 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:10.916063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zw265" podStartSLOduration=1.85760807 podStartE2EDuration="3.91604644s" totalImagesPullingTime="2.05843837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:10.914049742 +0000 UTC m=+452.487997088" watchObservedRunningTime="2026-07-07 17:48:10.91604644 +0000 UTC m=+452.489993780"
Jul 07 17:48:10.930971 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:10.930926    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9" podStartSLOduration=2.422472354 podStartE2EDuration="3.930910094s" totalImagesPullingTime="1.50843774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:10.929619495 +0000 UTC m=+452.503566836" watchObservedRunningTime="2026-07-07 17:48:10.930910094 +0000 UTC m=+452.504857412"
Jul 07 17:48:11.144779 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.144750    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:11.217021 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.216928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rmnm\" (UniqueName: \"kubernetes.io/projected/22f36e0f-3039-4900-83eb-0b4aa7bc395c-kube-api-access-7rmnm\") pod \"22f36e0f-3039-4900-83eb-0b4aa7bc395c\" (UID: \"22f36e0f-3039-4900-83eb-0b4aa7bc395c\") "
Jul 07 17:48:11.219187 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.219160    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f36e0f-3039-4900-83eb-0b4aa7bc395c-kube-api-access-7rmnm" (OuterVolumeSpecName: "kube-api-access-7rmnm") pod "22f36e0f-3039-4900-83eb-0b4aa7bc395c" (UID: "22f36e0f-3039-4900-83eb-0b4aa7bc395c"). InnerVolumeSpecName "kube-api-access-7rmnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:48:11.317719 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.317676    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rmnm\" (UniqueName: \"kubernetes.io/projected/22f36e0f-3039-4900-83eb-0b4aa7bc395c-kube-api-access-7rmnm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:48:11.907486 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.907451    2577 generic.go:356] "Generic (PLEG): container finished" podID="22f36e0f-3039-4900-83eb-0b4aa7bc395c" containerID="c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700" exitCode=0
Jul 07 17:48:11.907900 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.907509    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zw265"
Jul 07 17:48:11.907900 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.907531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zw265" event={"ID":"22f36e0f-3039-4900-83eb-0b4aa7bc395c","Type":"ContainerDied","Data":"c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700"}
Jul 07 17:48:11.907900 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.907597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zw265" event={"ID":"22f36e0f-3039-4900-83eb-0b4aa7bc395c","Type":"ContainerDied","Data":"0990e96301371fde6ed0a90dc445ebaae47f9a566c086ff2341542129ee7d295"}
Jul 07 17:48:11.907900 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.907617    2577 scope.go:117] "RemoveContainer" containerID="c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700"
Jul 07 17:48:11.916477 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.916462    2577 scope.go:117] "RemoveContainer" containerID="c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700"
Jul 07 17:48:11.916724 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:48:11.916707    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700\": container with ID starting with c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700 not found: ID does not exist" containerID="c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700"
Jul 07 17:48:11.916781 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.916737    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700"} err="failed to get container status \"c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700\": rpc error: code = NotFound desc = could not find container \"c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700\": container with ID starting with c2e2edc43bd9c5f1ed44ebcb2d8f0c1c4f06f6d67eb3ed966dfe1960a70ba700 not found: ID does not exist"
Jul 07 17:48:11.927489 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.927469    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zw265"]
Jul 07 17:48:11.931062 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:11.931043    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zw265"]
Jul 07 17:48:13.096334 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:13.096298    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f36e0f-3039-4900-83eb-0b4aa7bc395c" path="/var/lib/kubelet/pods/22f36e0f-3039-4900-83eb-0b4aa7bc395c/volumes"
Jul 07 17:48:18.241669 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:18.241587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:18.242103 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:18.241687    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:18.263714 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:18.263689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:18.960963 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:18.960935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-q2mk9"
Jul 07 17:48:39.874838 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.874804    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-975k5"]
Jul 07 17:48:39.875295 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.875101    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f36e0f-3039-4900-83eb-0b4aa7bc395c" containerName="registry-server"
Jul 07 17:48:39.875295 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.875134    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f36e0f-3039-4900-83eb-0b4aa7bc395c" containerName="registry-server"
Jul 07 17:48:39.875295 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.875188    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f36e0f-3039-4900-83eb-0b4aa7bc395c" containerName="registry-server"
Jul 07 17:48:39.879154 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.879139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:39.882748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.882726    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9w76g\""
Jul 07 17:48:39.897952 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:39.897929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-975k5"]
Jul 07 17:48:40.055655 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:40.055620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpcdv\" (UniqueName: \"kubernetes.io/projected/6b67ddc1-340e-4c64-aa94-5a6ad183421c-kube-api-access-kpcdv\") pod \"authorino-operator-75cc8d5cb7-975k5\" (UID: \"6b67ddc1-340e-4c64-aa94-5a6ad183421c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:40.156737 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:40.156650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpcdv\" (UniqueName: \"kubernetes.io/projected/6b67ddc1-340e-4c64-aa94-5a6ad183421c-kube-api-access-kpcdv\") pod \"authorino-operator-75cc8d5cb7-975k5\" (UID: \"6b67ddc1-340e-4c64-aa94-5a6ad183421c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:40.168078 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:40.168045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpcdv\" (UniqueName: \"kubernetes.io/projected/6b67ddc1-340e-4c64-aa94-5a6ad183421c-kube-api-access-kpcdv\") pod \"authorino-operator-75cc8d5cb7-975k5\" (UID: \"6b67ddc1-340e-4c64-aa94-5a6ad183421c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:40.188984 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:40.188956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:40.338091 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:40.338041    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-975k5"]
Jul 07 17:48:40.340776 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:48:40.340752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b67ddc1_340e_4c64_aa94_5a6ad183421c.slice/crio-f466b8507997055e1f7ce2293b1a8ebbc1809f4a983d1e18ac483a6b9953df38 WatchSource:0}: Error finding container f466b8507997055e1f7ce2293b1a8ebbc1809f4a983d1e18ac483a6b9953df38: Status 404 returned error can't find the container with id f466b8507997055e1f7ce2293b1a8ebbc1809f4a983d1e18ac483a6b9953df38
Jul 07 17:48:41.029841 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:41.029795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5" event={"ID":"6b67ddc1-340e-4c64-aa94-5a6ad183421c","Type":"ContainerStarted","Data":"f466b8507997055e1f7ce2293b1a8ebbc1809f4a983d1e18ac483a6b9953df38"}
Jul 07 17:48:42.574934 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.574904    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"]
Jul 07 17:48:42.579413 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.579395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:42.582824 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.582663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-w6dxw\""
Jul 07 17:48:42.585141 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.583149    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 07 17:48:42.602704 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.602681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"]
Jul 07 17:48:42.679746 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.679709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45x9\" (UniqueName: \"kubernetes.io/projected/08bdeede-a044-48b2-a90d-ed7a9fa6bf18-kube-api-access-m45x9\") pod \"dns-operator-controller-manager-864dd78c4c-bq267\" (UID: \"08bdeede-a044-48b2-a90d-ed7a9fa6bf18\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:42.781157 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.781100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45x9\" (UniqueName: \"kubernetes.io/projected/08bdeede-a044-48b2-a90d-ed7a9fa6bf18-kube-api-access-m45x9\") pod \"dns-operator-controller-manager-864dd78c4c-bq267\" (UID: \"08bdeede-a044-48b2-a90d-ed7a9fa6bf18\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:42.791223 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.791162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45x9\" (UniqueName: \"kubernetes.io/projected/08bdeede-a044-48b2-a90d-ed7a9fa6bf18-kube-api-access-m45x9\") pod \"dns-operator-controller-manager-864dd78c4c-bq267\" (UID: \"08bdeede-a044-48b2-a90d-ed7a9fa6bf18\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:42.895215 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:42.895178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:43.033874 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:43.033846    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"]
Jul 07 17:48:43.037031 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:48:43.037004    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bdeede_a044_48b2_a90d_ed7a9fa6bf18.slice/crio-9cec1a1853db97dbd15065967f5ccad0e5b41b8c07173d692fca3281278fe21e WatchSource:0}: Error finding container 9cec1a1853db97dbd15065967f5ccad0e5b41b8c07173d692fca3281278fe21e: Status 404 returned error can't find the container with id 9cec1a1853db97dbd15065967f5ccad0e5b41b8c07173d692fca3281278fe21e
Jul 07 17:48:43.041036 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:43.041011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5" event={"ID":"6b67ddc1-340e-4c64-aa94-5a6ad183421c","Type":"ContainerStarted","Data":"673c43d9462241fe78be2dafe980cb4aec920d7520e64d1db1c0d9a7e3bda9ac"}
Jul 07 17:48:43.041174 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:43.041161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:43.066758 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:43.066713    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5" podStartSLOduration=1.868071999 podStartE2EDuration="4.066698694s" totalImagesPullingTime="2.198626695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:43.065650154 +0000 UTC m=+484.639597496" watchObservedRunningTime="2026-07-07 17:48:43.066698694 +0000 UTC m=+484.640646035"
Jul 07 17:48:44.046624 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:44.046594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267" event={"ID":"08bdeede-a044-48b2-a90d-ed7a9fa6bf18","Type":"ContainerStarted","Data":"9cec1a1853db97dbd15065967f5ccad0e5b41b8c07173d692fca3281278fe21e"}
Jul 07 17:48:45.269841 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.269811    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"]
Jul 07 17:48:45.273281 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.273263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:48:45.276666 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.276635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6bprm\""
Jul 07 17:48:45.284477 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.284453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"]
Jul 07 17:48:45.405793 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.405707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jpdk\" (UniqueName: \"kubernetes.io/projected/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760-kube-api-access-9jpdk\") pod \"limitador-operator-controller-manager-859d7b78bd-tprls\" (UID: \"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:48:45.507076 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.507035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jpdk\" (UniqueName: \"kubernetes.io/projected/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760-kube-api-access-9jpdk\") pod \"limitador-operator-controller-manager-859d7b78bd-tprls\" (UID: \"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:48:45.516815 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.516791    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jpdk\" (UniqueName: \"kubernetes.io/projected/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760-kube-api-access-9jpdk\") pod \"limitador-operator-controller-manager-859d7b78bd-tprls\" (UID: \"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:48:45.583432 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.583395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:48:45.727678 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:45.727520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"]
Jul 07 17:48:45.730362 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:48:45.730335    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f28c22_2ce8_47a3_a2b9_7fda8d7cc760.slice/crio-3ede89aeb5ccaae53ad3173697dc5cafcb8878588eb5f3e4aca8cf927439f68d WatchSource:0}: Error finding container 3ede89aeb5ccaae53ad3173697dc5cafcb8878588eb5f3e4aca8cf927439f68d: Status 404 returned error can't find the container with id 3ede89aeb5ccaae53ad3173697dc5cafcb8878588eb5f3e4aca8cf927439f68d
Jul 07 17:48:46.055289 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:46.055255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls" event={"ID":"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760","Type":"ContainerStarted","Data":"3ede89aeb5ccaae53ad3173697dc5cafcb8878588eb5f3e4aca8cf927439f68d"}
Jul 07 17:48:46.056651 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:46.056621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267" event={"ID":"08bdeede-a044-48b2-a90d-ed7a9fa6bf18","Type":"ContainerStarted","Data":"03240c4c04406011d38578ad4f2e1c0585c5f68a326d845c4e5824576cb91f64"}
Jul 07 17:48:46.056793 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:46.056706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:46.095791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:46.095748    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267" podStartSLOduration=2.071384341 podStartE2EDuration="4.095732107s" totalImagesPullingTime="2.024347766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:46.094605103 +0000 UTC m=+487.668552443" watchObservedRunningTime="2026-07-07 17:48:46.095732107 +0000 UTC m=+487.669679448"
Jul 07 17:48:48.066704 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:48.066669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls" event={"ID":"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760","Type":"ContainerStarted","Data":"3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e"}
Jul 07 17:48:48.067063 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:48.066800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:48:48.088522 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:48.088473    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls" podStartSLOduration=1.617320663 podStartE2EDuration="3.088459243s" totalImagesPullingTime="1.47113858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:48:48.088198454 +0000 UTC m=+489.662145794" watchObservedRunningTime="2026-07-07 17:48:48.088459243 +0000 UTC m=+489.662406584"
Jul 07 17:48:54.048961 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:54.048931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-975k5"
Jul 07 17:48:57.063424 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:57.063400    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bq267"
Jul 07 17:48:59.073554 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:48:59.073529    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:08.508279 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.508243    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"]
Jul 07 17:49:08.511549 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.511532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.514221 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.514203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2zqnz\""
Jul 07 17:49:08.528846 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.528820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"]
Jul 07 17:49:08.590073 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.590045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46n4\" (UniqueName: \"kubernetes.io/projected/2d7627a8-aaf5-4d69-b155-c993f6c256b4-kube-api-access-c46n4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.590240 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.590101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7627a8-aaf5-4d69-b155-c993f6c256b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.691385 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.691339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7627a8-aaf5-4d69-b155-c993f6c256b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.691545 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.691420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46n4\" (UniqueName: \"kubernetes.io/projected/2d7627a8-aaf5-4d69-b155-c993f6c256b4-kube-api-access-c46n4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.691719 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.691700    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7627a8-aaf5-4d69-b155-c993f6c256b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.707253 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.707225    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46n4\" (UniqueName: \"kubernetes.io/projected/2d7627a8-aaf5-4d69-b155-c993f6c256b4-kube-api-access-c46n4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.822639 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.822548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:08.957125 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:08.957079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"]
Jul 07 17:49:08.960482 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:49:08.960449    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7627a8_aaf5_4d69_b155_c993f6c256b4.slice/crio-db3fd715f5a3c0f065ee0c6eb7cbe62989d7baf5e678483cac98938ad5ce9e94 WatchSource:0}: Error finding container db3fd715f5a3c0f065ee0c6eb7cbe62989d7baf5e678483cac98938ad5ce9e94: Status 404 returned error can't find the container with id db3fd715f5a3c0f065ee0c6eb7cbe62989d7baf5e678483cac98938ad5ce9e94
Jul 07 17:49:09.155582 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.155498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp" event={"ID":"2d7627a8-aaf5-4d69-b155-c993f6c256b4","Type":"ContainerStarted","Data":"db3fd715f5a3c0f065ee0c6eb7cbe62989d7baf5e678483cac98938ad5ce9e94"}
Jul 07 17:49:09.209522 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.209491    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"]
Jul 07 17:49:09.219812 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.219786    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"]
Jul 07 17:49:09.272163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.272129    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"]
Jul 07 17:49:09.272341 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.272317    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" containerName="manager" containerID="cri-o://3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e" gracePeriod=2
Jul 07 17:49:09.280528 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.280501    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"]
Jul 07 17:49:09.283631 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.283611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.295894 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.295869    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"]
Jul 07 17:49:09.316413 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:09.316376    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-tprls\": pods \"limitador-operator-controller-manager-859d7b78bd-tprls\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:09.318713 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:09.318677    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tprls\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:09.325818 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.325794    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"]
Jul 07 17:49:09.349022 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.348994    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"]
Jul 07 17:49:09.349529 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.349508    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" containerName="manager"
Jul 07 17:49:09.349584 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.349535    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" containerName="manager"
Jul 07 17:49:09.349635 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.349624    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" containerName="manager"
Jul 07 17:49:09.352614 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.352593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:09.370776 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:09.370751    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tprls\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:09.394523 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.394500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"]
Jul 07 17:49:09.397490 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.397467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlz7\" (UniqueName: \"kubernetes.io/projected/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-kube-api-access-gnlz7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-llpmq\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.397602 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.397568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-llpmq\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.397647 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.397606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92r62\" (UniqueName: \"kubernetes.io/projected/e788a6ab-bd40-4f77-a0b1-e2ffaf032f94-kube-api-access-92r62\") pod \"limitador-operator-controller-manager-859d7b78bd-2jhtr\" (UID: \"e788a6ab-bd40-4f77-a0b1-e2ffaf032f94\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:09.498748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.498710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-llpmq\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.498950 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.498772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r62\" (UniqueName: \"kubernetes.io/projected/e788a6ab-bd40-4f77-a0b1-e2ffaf032f94-kube-api-access-92r62\") pod \"limitador-operator-controller-manager-859d7b78bd-2jhtr\" (UID: \"e788a6ab-bd40-4f77-a0b1-e2ffaf032f94\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:09.498950 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.498810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlz7\" (UniqueName: \"kubernetes.io/projected/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-kube-api-access-gnlz7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-llpmq\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.499391 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.499184    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-llpmq\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.509688 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.509663    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r62\" (UniqueName: \"kubernetes.io/projected/e788a6ab-bd40-4f77-a0b1-e2ffaf032f94-kube-api-access-92r62\") pod \"limitador-operator-controller-manager-859d7b78bd-2jhtr\" (UID: \"e788a6ab-bd40-4f77-a0b1-e2ffaf032f94\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:09.510064 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.509688    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlz7\" (UniqueName: \"kubernetes.io/projected/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-kube-api-access-gnlz7\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-llpmq\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.515820 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.515799    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:09.518735 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:09.518703    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tprls\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:09.599912 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.599883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jpdk\" (UniqueName: \"kubernetes.io/projected/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760-kube-api-access-9jpdk\") pod \"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760\" (UID: \"64f28c22-2ce8-47a3-a2b9-7fda8d7cc760\") "
Jul 07 17:49:09.601866 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.601835    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760-kube-api-access-9jpdk" (OuterVolumeSpecName: "kube-api-access-9jpdk") pod "64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" (UID: "64f28c22-2ce8-47a3-a2b9-7fda8d7cc760"). InnerVolumeSpecName "kube-api-access-9jpdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:49:09.657034 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.656993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:09.664077 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.664056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:09.701322 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.701228    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jpdk\" (UniqueName: \"kubernetes.io/projected/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760-kube-api-access-9jpdk\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:49:09.817631 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.817603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"]
Jul 07 17:49:09.822089 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:49:09.822054    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadeceeaa_aa06_46e9_b8e7_476d68e55ab2.slice/crio-d589264722055336d214888f93033dd20ee1fa82b58ceaf563cce76b620adfad WatchSource:0}: Error finding container d589264722055336d214888f93033dd20ee1fa82b58ceaf563cce76b620adfad: Status 404 returned error can't find the container with id d589264722055336d214888f93033dd20ee1fa82b58ceaf563cce76b620adfad
Jul 07 17:49:09.835158 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:09.835132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"]
Jul 07 17:49:09.838853 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:49:09.838826    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode788a6ab_bd40_4f77_a0b1_e2ffaf032f94.slice/crio-0157bbd14c775d33c6afb8e09d23b8d99c5466d0fbaa5037c9d07971631b9781 WatchSource:0}: Error finding container 0157bbd14c775d33c6afb8e09d23b8d99c5466d0fbaa5037c9d07971631b9781: Status 404 returned error can't find the container with id 0157bbd14c775d33c6afb8e09d23b8d99c5466d0fbaa5037c9d07971631b9781
Jul 07 17:49:10.161537 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.161501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr" event={"ID":"e788a6ab-bd40-4f77-a0b1-e2ffaf032f94","Type":"ContainerStarted","Data":"c11dbf6069d134ed6b99411b3d805e5560d534146a57f93f03326af6ff3d290f"}
Jul 07 17:49:10.161745 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.161549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr" event={"ID":"e788a6ab-bd40-4f77-a0b1-e2ffaf032f94","Type":"ContainerStarted","Data":"0157bbd14c775d33c6afb8e09d23b8d99c5466d0fbaa5037c9d07971631b9781"}
Jul 07 17:49:10.161745 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.161635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:10.162840 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.162817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq" event={"ID":"adeceeaa-aa06-46e9-b8e7-476d68e55ab2","Type":"ContainerStarted","Data":"d589264722055336d214888f93033dd20ee1fa82b58ceaf563cce76b620adfad"}
Jul 07 17:49:10.163758 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:10.163732    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tprls\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:10.163971 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.163953    2577 generic.go:356] "Generic (PLEG): container finished" podID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" containerID="3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e" exitCode=0
Jul 07 17:49:10.164021 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.163984    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:10.164021 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.163995    2577 scope.go:117] "RemoveContainer" containerID="3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e"
Jul 07 17:49:10.172748 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.172734    2577 scope.go:117] "RemoveContainer" containerID="3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e"
Jul 07 17:49:10.172990 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:10.172971    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e\": container with ID starting with 3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e not found: ID does not exist" containerID="3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e"
Jul 07 17:49:10.173052 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.172997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e"} err="failed to get container status \"3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e\": rpc error: code = NotFound desc = could not find container \"3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e\": container with ID starting with 3158649b0cfdf3f1af72567ade78f4c52883e73f035c0fdf52ba268fc19e316e not found: ID does not exist"
Jul 07 17:49:10.188313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:10.188274    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr" podStartSLOduration=1.188260833 podStartE2EDuration="1.188260833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:49:10.187336078 +0000 UTC m=+511.761283423" watchObservedRunningTime="2026-07-07 17:49:10.188260833 +0000 UTC m=+511.762208245"
Jul 07 17:49:10.194682 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:10.194655    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-tprls\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tprls"
Jul 07 17:49:11.098803 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:11.098770    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f28c22-2ce8-47a3-a2b9-7fda8d7cc760" path="/var/lib/kubelet/pods/64f28c22-2ce8-47a3-a2b9-7fda8d7cc760/volumes"
Jul 07 17:49:14.188166 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.188125    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" containerName="manager" containerID="cri-o://463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300" gracePeriod=2
Jul 07 17:49:14.189608 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.189583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq" event={"ID":"adeceeaa-aa06-46e9-b8e7-476d68e55ab2","Type":"ContainerStarted","Data":"bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1"}
Jul 07 17:49:14.189753 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.189688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:14.225892 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.225832    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq" podStartSLOduration=1.4535947120000001 podStartE2EDuration="5.225810737s" totalImagesPullingTime="3.772216025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:49:14.22336279 +0000 UTC m=+515.797310131" watchObservedRunningTime="2026-07-07 17:49:14.225810737 +0000 UTC m=+515.799758080"
Jul 07 17:49:14.433616 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.433589    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:14.435705 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:14.435675    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:14.544635 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.544607    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c46n4\" (UniqueName: \"kubernetes.io/projected/2d7627a8-aaf5-4d69-b155-c993f6c256b4-kube-api-access-c46n4\") pod \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") "
Jul 07 17:49:14.544810 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.544643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7627a8-aaf5-4d69-b155-c993f6c256b4-extensions-socket-volume\") pod \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\" (UID: \"2d7627a8-aaf5-4d69-b155-c993f6c256b4\") "
Jul 07 17:49:14.544973 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.544950    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7627a8-aaf5-4d69-b155-c993f6c256b4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2d7627a8-aaf5-4d69-b155-c993f6c256b4" (UID: "2d7627a8-aaf5-4d69-b155-c993f6c256b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:49:14.546670 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.546645    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7627a8-aaf5-4d69-b155-c993f6c256b4-kube-api-access-c46n4" (OuterVolumeSpecName: "kube-api-access-c46n4") pod "2d7627a8-aaf5-4d69-b155-c993f6c256b4" (UID: "2d7627a8-aaf5-4d69-b155-c993f6c256b4"). InnerVolumeSpecName "kube-api-access-c46n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:49:14.645402 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.645365    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c46n4\" (UniqueName: \"kubernetes.io/projected/2d7627a8-aaf5-4d69-b155-c993f6c256b4-kube-api-access-c46n4\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:49:14.645402 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:14.645395    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d7627a8-aaf5-4d69-b155-c993f6c256b4-extensions-socket-volume\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:49:15.096807 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:15.096765    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" path="/var/lib/kubelet/pods/2d7627a8-aaf5-4d69-b155-c993f6c256b4/volumes"
Jul 07 17:49:15.194803 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:15.194764    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" containerID="463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300" exitCode=2
Jul 07 17:49:15.195247 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:15.194808    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:15.195247 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:15.194835    2577 scope.go:117] "RemoveContainer" containerID="463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300"
Jul 07 17:49:15.199637 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:15.199613    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6bzlp\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6bzlp"
Jul 07 17:49:15.202976 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:15.202953    2577 scope.go:117] "RemoveContainer" containerID="463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300"
Jul 07 17:49:15.208380 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:15.203706    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300\": container with ID starting with 463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300 not found: ID does not exist" containerID="463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300"
Jul 07 17:49:15.208380 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:15.203747    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300"} err="failed to get container status \"463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300\": rpc error: code = NotFound desc = could not find container \"463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300\": container with ID starting with 463ca4025ff334777882f2b34633c6a0df09a0b6f6068ef54afa68f59b3a5300 not found: ID does not exist"
Jul 07 17:49:21.172334 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:21.172307    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2jhtr"
Jul 07 17:49:25.197093 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:25.197063    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:37.537681 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.537651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"]
Jul 07 17:49:37.538100 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.537860    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq" podUID="adeceeaa-aa06-46e9-b8e7-476d68e55ab2" containerName="manager" containerID="cri-o://bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1" gracePeriod=10
Jul 07 17:49:37.790851 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.790794    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:37.841955 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.841928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-extensions-socket-volume\") pod \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") "
Jul 07 17:49:37.842096 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.841975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnlz7\" (UniqueName: \"kubernetes.io/projected/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-kube-api-access-gnlz7\") pod \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\" (UID: \"adeceeaa-aa06-46e9-b8e7-476d68e55ab2\") "
Jul 07 17:49:37.842406 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.842380    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "adeceeaa-aa06-46e9-b8e7-476d68e55ab2" (UID: "adeceeaa-aa06-46e9-b8e7-476d68e55ab2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 17:49:37.843960 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.843940    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-kube-api-access-gnlz7" (OuterVolumeSpecName: "kube-api-access-gnlz7") pod "adeceeaa-aa06-46e9-b8e7-476d68e55ab2" (UID: "adeceeaa-aa06-46e9-b8e7-476d68e55ab2"). InnerVolumeSpecName "kube-api-access-gnlz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:49:37.942461 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.942430    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-extensions-socket-volume\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:49:37.942461 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:37.942456    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnlz7\" (UniqueName: \"kubernetes.io/projected/adeceeaa-aa06-46e9-b8e7-476d68e55ab2-kube-api-access-gnlz7\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:49:38.306103 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.306070    2577 generic.go:356] "Generic (PLEG): container finished" podID="adeceeaa-aa06-46e9-b8e7-476d68e55ab2" containerID="bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1" exitCode=0
Jul 07 17:49:38.306255 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.306125    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"
Jul 07 17:49:38.306255 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.306145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq" event={"ID":"adeceeaa-aa06-46e9-b8e7-476d68e55ab2","Type":"ContainerDied","Data":"bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1"}
Jul 07 17:49:38.306255 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.306199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq" event={"ID":"adeceeaa-aa06-46e9-b8e7-476d68e55ab2","Type":"ContainerDied","Data":"d589264722055336d214888f93033dd20ee1fa82b58ceaf563cce76b620adfad"}
Jul 07 17:49:38.306255 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.306223    2577 scope.go:117] "RemoveContainer" containerID="bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1"
Jul 07 17:49:38.314768 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.314750    2577 scope.go:117] "RemoveContainer" containerID="bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1"
Jul 07 17:49:38.315033 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:49:38.315015    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1\": container with ID starting with bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1 not found: ID does not exist" containerID="bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1"
Jul 07 17:49:38.315083 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.315042    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1"} err="failed to get container status \"bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1\": rpc error: code = NotFound desc = could not find container \"bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1\": container with ID starting with bd3eefadb0cb996d656c07f5960bb6f6b6fb2189a123d98288fa081b5d9808c1 not found: ID does not exist"
Jul 07 17:49:38.359765 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.359739    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"]
Jul 07 17:49:38.361060 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:38.361040    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-llpmq"]
Jul 07 17:49:39.096529 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:39.096498    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adeceeaa-aa06-46e9-b8e7-476d68e55ab2" path="/var/lib/kubelet/pods/adeceeaa-aa06-46e9-b8e7-476d68e55ab2/volumes"
Jul 07 17:49:42.154602 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154574    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"]
Jul 07 17:49:42.155167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154884    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adeceeaa-aa06-46e9-b8e7-476d68e55ab2" containerName="manager"
Jul 07 17:49:42.155167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154896    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="adeceeaa-aa06-46e9-b8e7-476d68e55ab2" containerName="manager"
Jul 07 17:49:42.155167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154910    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" containerName="manager"
Jul 07 17:49:42.155167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154915    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" containerName="manager"
Jul 07 17:49:42.155167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154986    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="adeceeaa-aa06-46e9-b8e7-476d68e55ab2" containerName="manager"
Jul 07 17:49:42.155167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.154996    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7627a8-aaf5-4d69-b155-c993f6c256b4" containerName="manager"
Jul 07 17:49:42.165429 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.165400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.167844 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.167815    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bn6ll\""
Jul 07 17:49:42.173426 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"]
Jul 07 17:49:42.173777 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.173777 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.173928 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.173928 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7t9\" (UniqueName: \"kubernetes.io/projected/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-kube-api-access-7r7t9\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.173928 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.173928 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.173928 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.174197 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.173978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.174197 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.174044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.274802 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.274770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.274802 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.274805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275053 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.274825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7t9\" (UniqueName: \"kubernetes.io/projected/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-kube-api-access-7r7t9\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275053 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.274958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275053 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275053 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275387 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275387 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275387 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275529 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275494    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275588 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275588 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275562    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.275689 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.275594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.276041 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.276018    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.277163 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.277143    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.277427 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.277407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.283704 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.283683    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7t9\" (UniqueName: \"kubernetes.io/projected/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-kube-api-access-7r7t9\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.284084 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.284067    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77e8b1fd-f9f6-41ea-aba7-b893af5fc41f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-stz9d\" (UID: \"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.477160 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.477049    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:42.630695 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:49:42.630667    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e8b1fd_f9f6_41ea_aba7_b893af5fc41f.slice/crio-b27e4e826b628154b8745920fb11c9382df1ece5c4fd320d8ea07bc5da0e1100 WatchSource:0}: Error finding container b27e4e826b628154b8745920fb11c9382df1ece5c4fd320d8ea07bc5da0e1100: Status 404 returned error can't find the container with id b27e4e826b628154b8745920fb11c9382df1ece5c4fd320d8ea07bc5da0e1100
Jul 07 17:49:42.633178 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.633142    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:49:42.633291 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.633229    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:49:42.633291 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.633275    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:49:42.635167 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:42.635147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"]
Jul 07 17:49:43.328685 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:43.328654    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" event={"ID":"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f","Type":"ContainerStarted","Data":"abe168e02071e2742a7628293cb1de3076f8df495f792b98dcc00d95797e6b85"}
Jul 07 17:49:43.329184 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:43.328713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" event={"ID":"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f","Type":"ContainerStarted","Data":"b27e4e826b628154b8745920fb11c9382df1ece5c4fd320d8ea07bc5da0e1100"}
Jul 07 17:49:43.349323 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:43.349227    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" podStartSLOduration=1.349183892 podStartE2EDuration="1.349183892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:49:43.348883567 +0000 UTC m=+544.922830908" watchObservedRunningTime="2026-07-07 17:49:43.349183892 +0000 UTC m=+544.923131233"
Jul 07 17:49:43.477645 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:43.477613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:43.482190 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:43.482166    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:44.333558 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:44.333530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:44.334371 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:44.334349    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:49:57.703045 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.703010    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:49:57.710550 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.710521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:57.713063 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.713037    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 17:49:57.713063 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.713045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ptkw2\""
Jul 07 17:49:57.716553 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.716527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:49:57.748286 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.748257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:49:57.802462 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.802429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvgv\" (UniqueName: \"kubernetes.io/projected/aff70cc8-445d-4a9e-bed3-1ca0fce32221-kube-api-access-dqvgv\") pod \"limitador-limitador-54d6d988d5-btrpx\" (UID: \"aff70cc8-445d-4a9e-bed3-1ca0fce32221\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:57.802601 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.802504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aff70cc8-445d-4a9e-bed3-1ca0fce32221-config-file\") pod \"limitador-limitador-54d6d988d5-btrpx\" (UID: \"aff70cc8-445d-4a9e-bed3-1ca0fce32221\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:57.903926 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.903874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aff70cc8-445d-4a9e-bed3-1ca0fce32221-config-file\") pod \"limitador-limitador-54d6d988d5-btrpx\" (UID: \"aff70cc8-445d-4a9e-bed3-1ca0fce32221\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:57.904103 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.903947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvgv\" (UniqueName: \"kubernetes.io/projected/aff70cc8-445d-4a9e-bed3-1ca0fce32221-kube-api-access-dqvgv\") pod \"limitador-limitador-54d6d988d5-btrpx\" (UID: \"aff70cc8-445d-4a9e-bed3-1ca0fce32221\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:57.904534 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.904513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aff70cc8-445d-4a9e-bed3-1ca0fce32221-config-file\") pod \"limitador-limitador-54d6d988d5-btrpx\" (UID: \"aff70cc8-445d-4a9e-bed3-1ca0fce32221\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:57.912619 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:57.912591    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvgv\" (UniqueName: \"kubernetes.io/projected/aff70cc8-445d-4a9e-bed3-1ca0fce32221-kube-api-access-dqvgv\") pod \"limitador-limitador-54d6d988d5-btrpx\" (UID: \"aff70cc8-445d-4a9e-bed3-1ca0fce32221\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:58.021703 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:58.021674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:49:58.157415 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:58.157385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:49:58.160981 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:49:58.160958    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff70cc8_445d_4a9e_bed3_1ca0fce32221.slice/crio-b6ca1b5bc50b98ee01356a1ecf2b2c8571e631c2df53e5c430ee7f88ca9a2009 WatchSource:0}: Error finding container b6ca1b5bc50b98ee01356a1ecf2b2c8571e631c2df53e5c430ee7f88ca9a2009: Status 404 returned error can't find the container with id b6ca1b5bc50b98ee01356a1ecf2b2c8571e631c2df53e5c430ee7f88ca9a2009
Jul 07 17:49:58.400003 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:49:58.399925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx" event={"ID":"aff70cc8-445d-4a9e-bed3-1ca0fce32221","Type":"ContainerStarted","Data":"b6ca1b5bc50b98ee01356a1ecf2b2c8571e631c2df53e5c430ee7f88ca9a2009"}
Jul 07 17:50:01.417000 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:01.416968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx" event={"ID":"aff70cc8-445d-4a9e-bed3-1ca0fce32221","Type":"ContainerStarted","Data":"f1ed71af941658b4258846472d15ca376c6a0245a22ef676a92adc32cad49030"}
Jul 07 17:50:01.417488 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:01.417026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:50:01.433967 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:01.433925    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx" podStartSLOduration=1.804580512 podStartE2EDuration="4.433905987s" totalImagesPullingTime="2.629325475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:50:01.432864922 +0000 UTC m=+563.006812264" watchObservedRunningTime="2026-07-07 17:50:01.433905987 +0000 UTC m=+563.007853329"
Jul 07 17:50:12.422808 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:12.422779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-btrpx"
Jul 07 17:50:28.766131 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.766080    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75b965877d-qrp47"]
Jul 07 17:50:28.769588 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.769572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:28.777700 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.777679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rzbhw\""
Jul 07 17:50:28.778610 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.778583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 17:50:28.794794 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.794768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b965877d-qrp47"]
Jul 07 17:50:28.867804 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.867774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5fd\" (UniqueName: \"kubernetes.io/projected/df842539-5934-4723-986b-858bf0d1b4f7-kube-api-access-tt5fd\") pod \"authorino-75b965877d-qrp47\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") " pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:28.867962 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.867835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df842539-5934-4723-986b-858bf0d1b4f7-tls-cert\") pod \"authorino-75b965877d-qrp47\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") " pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:28.969050 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.969022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5fd\" (UniqueName: \"kubernetes.io/projected/df842539-5934-4723-986b-858bf0d1b4f7-kube-api-access-tt5fd\") pod \"authorino-75b965877d-qrp47\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") " pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:28.969217 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.969096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df842539-5934-4723-986b-858bf0d1b4f7-tls-cert\") pod \"authorino-75b965877d-qrp47\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") " pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:28.971466 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.971445    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df842539-5934-4723-986b-858bf0d1b4f7-tls-cert\") pod \"authorino-75b965877d-qrp47\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") " pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:28.979661 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:28.979636    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5fd\" (UniqueName: \"kubernetes.io/projected/df842539-5934-4723-986b-858bf0d1b4f7-kube-api-access-tt5fd\") pod \"authorino-75b965877d-qrp47\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") " pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:29.078315 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:29.078246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:50:29.228787 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:29.228249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b965877d-qrp47"]
Jul 07 17:50:29.237171 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:50:29.237137    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf842539_5934_4723_986b_858bf0d1b4f7.slice/crio-c4850c65accc02406eab2d581cca90cbb58da155ec4c6730e2aa24ba91294b84 WatchSource:0}: Error finding container c4850c65accc02406eab2d581cca90cbb58da155ec4c6730e2aa24ba91294b84: Status 404 returned error can't find the container with id c4850c65accc02406eab2d581cca90cbb58da155ec4c6730e2aa24ba91294b84
Jul 07 17:50:29.542632 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:29.542586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b965877d-qrp47" event={"ID":"df842539-5934-4723-986b-858bf0d1b4f7","Type":"ContainerStarted","Data":"c4850c65accc02406eab2d581cca90cbb58da155ec4c6730e2aa24ba91294b84"}
Jul 07 17:50:32.562493 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:32.562411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b965877d-qrp47" event={"ID":"df842539-5934-4723-986b-858bf0d1b4f7","Type":"ContainerStarted","Data":"5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86"}
Jul 07 17:50:32.580564 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:32.580519    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75b965877d-qrp47" podStartSLOduration=1.551405633 podStartE2EDuration="4.580504414s" totalImagesPullingTime="3.029098781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:50:32.578728235 +0000 UTC m=+594.152675576" watchObservedRunningTime="2026-07-07 17:50:32.580504414 +0000 UTC m=+594.154451758"
Jul 07 17:50:38.887816 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:38.887784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:50:38.888281 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:50:38.888262    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:51:12.837865 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:12.837832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:51:20.788093 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:20.788059    2577 generic.go:356] "Generic (PLEG): container finished" podID="77e8b1fd-f9f6-41ea-aba7-b893af5fc41f" containerID="abe168e02071e2742a7628293cb1de3076f8df495f792b98dcc00d95797e6b85" exitCode=0
Jul 07 17:51:20.788530 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:20.788133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" event={"ID":"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f","Type":"ContainerDied","Data":"abe168e02071e2742a7628293cb1de3076f8df495f792b98dcc00d95797e6b85"}
Jul 07 17:51:20.788530 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:20.788522    2577 scope.go:117] "RemoveContainer" containerID="abe168e02071e2742a7628293cb1de3076f8df495f792b98dcc00d95797e6b85"
Jul 07 17:51:20.789468 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:20.789444    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:51:20.789580 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:20.789509    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:51:20.789580 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:20.789539    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 17:51:21.794649 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:21.794615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" event={"ID":"77e8b1fd-f9f6-41ea-aba7-b893af5fc41f","Type":"ContainerStarted","Data":"a4aa4e2c61163e98b4df2a7b6ffc67f536fc5fc34bdefd5e5131f9514eb4ad4f"}
Jul 07 17:51:22.477574 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:22.477534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:51:22.477856 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:22.477832    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-stz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 07 17:51:22.477919 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:22.477894    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" podUID="77e8b1fd-f9f6-41ea-aba7-b893af5fc41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 07 17:51:23.478432 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:23.478393    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-stz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 07 17:51:23.478829 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:23.478486    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" podUID="77e8b1fd-f9f6-41ea-aba7-b893af5fc41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 07 17:51:24.478161 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:24.478100    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-stz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 07 17:51:24.478338 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:24.478191    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" podUID="77e8b1fd-f9f6-41ea-aba7-b893af5fc41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 07 17:51:25.478101 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:25.478063    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-stz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 07 17:51:25.478486 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:25.478141    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d" podUID="77e8b1fd-f9f6-41ea-aba7-b893af5fc41f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 07 17:51:26.481393 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:26.481362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:51:26.481812 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:26.481663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:51:26.482421 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:26.482404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-stz9d"
Jul 07 17:51:36.210039 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:36.209995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:51:59.795394 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:51:59.795360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:52:06.197652 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:06.197619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:52:10.502053 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:10.502017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:52:24.785675 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.785639    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"]
Jul 07 17:52:24.788121 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.788088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:24.791426 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.791385    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xtlc6\""
Jul 07 17:52:24.791426 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.791416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 17:52:24.791614 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.791447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 17:52:24.791614 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.791521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 07 17:52:24.800182 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.800160    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"]
Jul 07 17:52:24.957927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.957889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:24.957927 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.957938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b58f8a-15a7-411e-8e0d-9cf7f548355e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:24.958313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.958142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:24.958313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.958189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:24.958313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.958216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:24.958313 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:24.958275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbl5\" (UniqueName: \"kubernetes.io/projected/69b58f8a-15a7-411e-8e0d-9cf7f548355e-kube-api-access-wpbl5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059354 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059354 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059577 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbl5\" (UniqueName: \"kubernetes.io/projected/69b58f8a-15a7-411e-8e0d-9cf7f548355e-kube-api-access-wpbl5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059577 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059577 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b58f8a-15a7-411e-8e0d-9cf7f548355e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059577 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059768 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059811 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.059845 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.059836    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.061631 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.061612    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b58f8a-15a7-411e-8e0d-9cf7f548355e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.061753 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.061736    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b58f8a-15a7-411e-8e0d-9cf7f548355e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.067025 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.066993    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbl5\" (UniqueName: \"kubernetes.io/projected/69b58f8a-15a7-411e-8e0d-9cf7f548355e-kube-api-access-wpbl5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jwbxw\" (UID: \"69b58f8a-15a7-411e-8e0d-9cf7f548355e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.098257 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.098230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:25.231905 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.231785    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"]
Jul 07 17:52:25.235322 ip-10-0-138-144 kubenswrapper[2577]: W0707 17:52:25.235288    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b58f8a_15a7_411e_8e0d_9cf7f548355e.slice/crio-92c77e35c258aee9c12ae2c85149d0f3139fd484a31fa3679d649a8eda5ff32a WatchSource:0}: Error finding container 92c77e35c258aee9c12ae2c85149d0f3139fd484a31fa3679d649a8eda5ff32a: Status 404 returned error can't find the container with id 92c77e35c258aee9c12ae2c85149d0f3139fd484a31fa3679d649a8eda5ff32a
Jul 07 17:52:25.236624 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:25.236609    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 17:52:26.085185 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:26.085140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw" event={"ID":"69b58f8a-15a7-411e-8e0d-9cf7f548355e","Type":"ContainerStarted","Data":"92c77e35c258aee9c12ae2c85149d0f3139fd484a31fa3679d649a8eda5ff32a"}
Jul 07 17:52:27.064527 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:27.064493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:52:27.097004 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:27.096973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw" event={"ID":"69b58f8a-15a7-411e-8e0d-9cf7f548355e","Type":"ContainerStarted","Data":"04c95b21c31dd91119d973215f6eb3bcdfa2e3f7a3588d5264f7b32cc767f5cd"}
Jul 07 17:52:27.121085 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:27.121038    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw" podStartSLOduration=2.128239663 podStartE2EDuration="3.121023298s" totalImagesPullingTime="992.783635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:52:27.119758924 +0000 UTC m=+708.693706267" watchObservedRunningTime="2026-07-07 17:52:27.121023298 +0000 UTC m=+708.694970638"
Jul 07 17:52:33.319300 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:33.319267    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:52:35.099333 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:35.099296    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:35.099722 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:35.099360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:35.112103 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:35.112074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:52:35.139614 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:52:35.139588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jwbxw"
Jul 07 17:53:57.756901 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:57.756867    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75b965877d-qrp47"]
Jul 07 17:53:57.757472 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:57.757064    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75b965877d-qrp47" podUID="df842539-5934-4723-986b-858bf0d1b4f7" containerName="authorino" containerID="cri-o://5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86" gracePeriod=30
Jul 07 17:53:58.012218 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.012158    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:53:58.181183 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.181150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df842539-5934-4723-986b-858bf0d1b4f7-tls-cert\") pod \"df842539-5934-4723-986b-858bf0d1b4f7\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") "
Jul 07 17:53:58.181349 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.181196    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt5fd\" (UniqueName: \"kubernetes.io/projected/df842539-5934-4723-986b-858bf0d1b4f7-kube-api-access-tt5fd\") pod \"df842539-5934-4723-986b-858bf0d1b4f7\" (UID: \"df842539-5934-4723-986b-858bf0d1b4f7\") "
Jul 07 17:53:58.183319 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.183288    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df842539-5934-4723-986b-858bf0d1b4f7-kube-api-access-tt5fd" (OuterVolumeSpecName: "kube-api-access-tt5fd") pod "df842539-5934-4723-986b-858bf0d1b4f7" (UID: "df842539-5934-4723-986b-858bf0d1b4f7"). InnerVolumeSpecName "kube-api-access-tt5fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 17:53:58.191054 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.191031    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df842539-5934-4723-986b-858bf0d1b4f7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "df842539-5934-4723-986b-858bf0d1b4f7" (UID: "df842539-5934-4723-986b-858bf0d1b4f7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 17:53:58.281940 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.281865    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/df842539-5934-4723-986b-858bf0d1b4f7-tls-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:53:58.281940 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.281893    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5fd\" (UniqueName: \"kubernetes.io/projected/df842539-5934-4723-986b-858bf0d1b4f7-kube-api-access-tt5fd\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 17:53:58.496676 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.496644    2577 generic.go:356] "Generic (PLEG): container finished" podID="df842539-5934-4723-986b-858bf0d1b4f7" containerID="5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86" exitCode=0
Jul 07 17:53:58.496839 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.496687    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b965877d-qrp47"
Jul 07 17:53:58.496839 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.496710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b965877d-qrp47" event={"ID":"df842539-5934-4723-986b-858bf0d1b4f7","Type":"ContainerDied","Data":"5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86"}
Jul 07 17:53:58.496839 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.496741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b965877d-qrp47" event={"ID":"df842539-5934-4723-986b-858bf0d1b4f7","Type":"ContainerDied","Data":"c4850c65accc02406eab2d581cca90cbb58da155ec4c6730e2aa24ba91294b84"}
Jul 07 17:53:58.496839 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.496757    2577 scope.go:117] "RemoveContainer" containerID="5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86"
Jul 07 17:53:58.507018 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.506999    2577 scope.go:117] "RemoveContainer" containerID="5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86"
Jul 07 17:53:58.507309 ip-10-0-138-144 kubenswrapper[2577]: E0707 17:53:58.507286    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86\": container with ID starting with 5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86 not found: ID does not exist" containerID="5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86"
Jul 07 17:53:58.507368 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.507326    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86"} err="failed to get container status \"5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86\": rpc error: code = NotFound desc = could not find container \"5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86\": container with ID starting with 5a57740890a6a748043b2af0d02bad7106aa1857b9b36925cc53b8c774ccbc86 not found: ID does not exist"
Jul 07 17:53:58.519317 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.519292    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75b965877d-qrp47"]
Jul 07 17:53:58.526067 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:58.526045    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75b965877d-qrp47"]
Jul 07 17:53:59.096897 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:53:59.096858    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df842539-5934-4723-986b-858bf0d1b4f7" path="/var/lib/kubelet/pods/df842539-5934-4723-986b-858bf0d1b4f7/volumes"
Jul 07 17:54:33.916374 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:54:33.916324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:54:43.198643 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:54:43.198597    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:54:54.497246 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:54:54.497214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:55:03.002697 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:55:03.002656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:55:13.604373 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:55:13.604340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:55:38.919416 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:55:38.919387    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:55:38.919884 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:55:38.919750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 17:55:43.031420 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:55:43.031380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:55:54.904868 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:55:54.904829    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:56:27.703544 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:56:27.703505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:56:36.610034 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:56:36.609990    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:56:46.913791 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:56:46.913747    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:56:54.012168 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:56:54.012130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:57:03.610929 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:57:03.610825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:57:15.302694 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:57:15.302660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:57:26.394790 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:57:26.394753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:57:36.511654 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:57:36.511617    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:58:02.107358 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:58:02.107325    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:58:11.310015 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:58:11.309977    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:58:34.906941 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:58:34.906898    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:58:45.468714 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:58:45.468677    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:59:01.937690 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:59:01.937655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:59:12.907292 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:59:12.907254    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:59:23.406008 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:59:23.405974    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:59:32.712378 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:59:32.712340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 17:59:41.100521 ip-10-0-138-144 kubenswrapper[2577]: I0707 17:59:41.100489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:00:00.149074 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.148997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724120-kw48k"]
Jul 07 18:00:00.149555 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.149382    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df842539-5934-4723-986b-858bf0d1b4f7" containerName="authorino"
Jul 07 18:00:00.149555 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.149396    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="df842539-5934-4723-986b-858bf0d1b4f7" containerName="authorino"
Jul 07 18:00:00.149555 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.149459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="df842539-5934-4723-986b-858bf0d1b4f7" containerName="authorino"
Jul 07 18:00:00.152319 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.152303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:00.154581 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.154560    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-bmbdf\""
Jul 07 18:00:00.168220 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.168197    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724120-kw48k"]
Jul 07 18:00:00.280324 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.280287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7d7x\" (UniqueName: \"kubernetes.io/projected/b6c127e6-05d7-4ab5-8982-9023281d28dc-kube-api-access-q7d7x\") pod \"maas-api-key-cleanup-29724120-kw48k\" (UID: \"b6c127e6-05d7-4ab5-8982-9023281d28dc\") " pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:00.382176 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.382135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7d7x\" (UniqueName: \"kubernetes.io/projected/b6c127e6-05d7-4ab5-8982-9023281d28dc-kube-api-access-q7d7x\") pod \"maas-api-key-cleanup-29724120-kw48k\" (UID: \"b6c127e6-05d7-4ab5-8982-9023281d28dc\") " pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:00.392372 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.392344    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7d7x\" (UniqueName: \"kubernetes.io/projected/b6c127e6-05d7-4ab5-8982-9023281d28dc-kube-api-access-q7d7x\") pod \"maas-api-key-cleanup-29724120-kw48k\" (UID: \"b6c127e6-05d7-4ab5-8982-9023281d28dc\") " pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:00.462616 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.462528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:00.593093 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.593068    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724120-kw48k"]
Jul 07 18:00:00.595834 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:00:00.595810    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c127e6_05d7_4ab5_8982_9023281d28dc.slice/crio-25c4f1a4d15bdd135814e397e9f7558bc486332c115deebdba8c2cf65a7f9993 WatchSource:0}: Error finding container 25c4f1a4d15bdd135814e397e9f7558bc486332c115deebdba8c2cf65a7f9993: Status 404 returned error can't find the container with id 25c4f1a4d15bdd135814e397e9f7558bc486332c115deebdba8c2cf65a7f9993
Jul 07 18:00:00.597627 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:00.597607    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:00:01.153842 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:01.153805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k" event={"ID":"b6c127e6-05d7-4ab5-8982-9023281d28dc","Type":"ContainerStarted","Data":"25c4f1a4d15bdd135814e397e9f7558bc486332c115deebdba8c2cf65a7f9993"}
Jul 07 18:00:08.188277 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:08.188229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k" event={"ID":"b6c127e6-05d7-4ab5-8982-9023281d28dc","Type":"ContainerStarted","Data":"9d9cbe85512e44ab87ee3dec4de7dd9c4bb397af1445a9798c4dc6dce4a86965"}
Jul 07 18:00:08.218328 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:08.218280    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k" podStartSLOduration=1.513116463 podStartE2EDuration="8.218265441s" totalImagesPullingTime="6.705148978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:08.215840606 +0000 UTC m=+1169.789787958" watchObservedRunningTime="2026-07-07 18:00:08.218265441 +0000 UTC m=+1169.792212782"
Jul 07 18:00:09.194015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:09.193980    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6c127e6-05d7-4ab5-8982-9023281d28dc" containerID="9d9cbe85512e44ab87ee3dec4de7dd9c4bb397af1445a9798c4dc6dce4a86965" exitCode=0
Jul 07 18:00:09.194477 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:09.194098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k" event={"ID":"b6c127e6-05d7-4ab5-8982-9023281d28dc","Type":"ContainerDied","Data":"9d9cbe85512e44ab87ee3dec4de7dd9c4bb397af1445a9798c4dc6dce4a86965"}
Jul 07 18:00:10.333698 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:10.333674    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:10.469787 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:10.469674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7d7x\" (UniqueName: \"kubernetes.io/projected/b6c127e6-05d7-4ab5-8982-9023281d28dc-kube-api-access-q7d7x\") pod \"b6c127e6-05d7-4ab5-8982-9023281d28dc\" (UID: \"b6c127e6-05d7-4ab5-8982-9023281d28dc\") "
Jul 07 18:00:10.471890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:10.471864    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c127e6-05d7-4ab5-8982-9023281d28dc-kube-api-access-q7d7x" (OuterVolumeSpecName: "kube-api-access-q7d7x") pod "b6c127e6-05d7-4ab5-8982-9023281d28dc" (UID: "b6c127e6-05d7-4ab5-8982-9023281d28dc"). InnerVolumeSpecName "kube-api-access-q7d7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:00:10.570351 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:10.570300    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7d7x\" (UniqueName: \"kubernetes.io/projected/b6c127e6-05d7-4ab5-8982-9023281d28dc-kube-api-access-q7d7x\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:00:11.204191 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:11.204164    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k"
Jul 07 18:00:11.204191 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:11.204179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724120-kw48k" event={"ID":"b6c127e6-05d7-4ab5-8982-9023281d28dc","Type":"ContainerDied","Data":"25c4f1a4d15bdd135814e397e9f7558bc486332c115deebdba8c2cf65a7f9993"}
Jul 07 18:00:11.204396 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:11.204223    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c4f1a4d15bdd135814e397e9f7558bc486332c115deebdba8c2cf65a7f9993"
Jul 07 18:00:38.946146 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:38.946098    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:00:38.947235 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:00:38.947199    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:01:31.523037 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:01:31.522995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:01:42.114470 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:01:42.114435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:01:51.826180 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:01:51.826142    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:02:03.809280 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:02:03.809247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:02:15.917751 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:02:15.917715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:02:24.922756 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:02:24.922719    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:02:35.109372 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:02:35.109337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:02:45.210801 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:02:45.210766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:02:57.107369 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:02:57.107336    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:03:06.612064 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:03:06.611985    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:03:18.304872 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:03:18.304831    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:03:43.301368 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:03:43.301327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:03:50.404032 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:03:50.403998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:03:59.102494 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:03:59.102460    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:04:11.406228 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:04:11.406184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:04:26.103941 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:04:26.103902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:04:36.902437 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:04:36.902402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:04:45.802778 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:04:45.802743    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:04:55.702271 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:04:55.702232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:05:06.409812 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:06.409778    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:05:16.103008 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:16.102957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:05:29.399361 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:29.399327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:05:38.973914 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:38.973885    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:05:38.975780 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:38.975762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:05:39.020672 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:39.020638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:05:48.329020 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:48.328986    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:05:57.711009 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:05:57.710967    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:06:10.103664 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:06:10.103584    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:06:20.210241 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:06:20.210208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:06:32.814708 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:06:32.814672    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:06:45.312430 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:06:45.312391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:06:55.413560 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:06:55.413522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:07:07.499355 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:07.499314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:07:18.102420 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:18.102381    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:07:28.100035 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.100004    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"]
Jul 07 18:07:28.100382 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.100352    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c127e6-05d7-4ab5-8982-9023281d28dc" containerName="cleanup"
Jul 07 18:07:28.100382 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.100363    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c127e6-05d7-4ab5-8982-9023281d28dc" containerName="cleanup"
Jul 07 18:07:28.100451 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.100417    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c127e6-05d7-4ab5-8982-9023281d28dc" containerName="cleanup"
Jul 07 18:07:28.103478 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.103463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.106710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.106688    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-afd9b7ef-openshift-default-dockercfg-bgxvz\""
Jul 07 18:07:28.126562 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.126537    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"]
Jul 07 18:07:28.186618 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-certs\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186775 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d711e9-593e-4801-b619-3041e1bb17d5-istiod-ca-cert\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186775 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-envoy\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186775 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-socket\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-credential-socket\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d711e9-593e-4801-b619-3041e1bb17d5-istio-podinfo\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9z88\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-kube-api-access-z9z88\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.186972 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-data\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.187015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.186971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-istio-token\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.207622 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.207588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:07:28.287445 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-envoy\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287628 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-socket\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287628 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-credential-socket\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287628 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d711e9-593e-4801-b619-3041e1bb17d5-istio-podinfo\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287628 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9z88\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-kube-api-access-z9z88\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287628 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-data\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287905 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-istio-token\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287905 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-certs\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.287905 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d711e9-593e-4801-b619-3041e1bb17d5-istiod-ca-cert\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.288067 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-socket\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.288067 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.287942    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-credential-socket\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.288353 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.288306    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-certs\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.288467 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.288436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d711e9-593e-4801-b619-3041e1bb17d5-istiod-ca-cert\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.288554 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.288531    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-data\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.290172 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.290149    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-envoy\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.290258 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.290171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d711e9-593e-4801-b619-3041e1bb17d5-istio-podinfo\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.295140 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.295121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-istio-token\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.296053 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.296031    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9z88\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-kube-api-access-z9z88\") pod \"e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") " pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.413406 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.413320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:28.551051 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.551024    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"]
Jul 07 18:07:28.552999 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:07:28.552970    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d711e9_593e_4801_b619_3041e1bb17d5.slice/crio-cc0b2aed9b914db0a148ab28c5ff454aea51cc9dc620a789f283b3f5cb3ed974 WatchSource:0}: Error finding container cc0b2aed9b914db0a148ab28c5ff454aea51cc9dc620a789f283b3f5cb3ed974: Status 404 returned error can't find the container with id cc0b2aed9b914db0a148ab28c5ff454aea51cc9dc620a789f283b3f5cb3ed974
Jul 07 18:07:28.555058 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.555031    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:28.555148 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.555093    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:28.555198 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:28.555147    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:29.178822 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:29.178786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg" event={"ID":"51d711e9-593e-4801-b619-3041e1bb17d5","Type":"ContainerStarted","Data":"3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627"}
Jul 07 18:07:29.179234 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:29.178842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg" event={"ID":"51d711e9-593e-4801-b619-3041e1bb17d5","Type":"ContainerStarted","Data":"cc0b2aed9b914db0a148ab28c5ff454aea51cc9dc620a789f283b3f5cb3ed974"}
Jul 07 18:07:29.199789 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:29.199743    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg" podStartSLOduration=1.1997241 podStartE2EDuration="1.1997241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:07:29.198001141 +0000 UTC m=+1610.771948482" watchObservedRunningTime="2026-07-07 18:07:29.1997241 +0000 UTC m=+1610.773671441"
Jul 07 18:07:29.414373 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:29.414337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:29.419288 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:29.419262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:30.183205 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:30.183128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:30.184153 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:30.184138    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:34.640843 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:34.640805    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"]
Jul 07 18:07:34.641651 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:34.641609    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg" podUID="51d711e9-593e-4801-b619-3041e1bb17d5" containerName="istio-proxy" containerID="cri-o://3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627" gracePeriod=30
Jul 07 18:07:41.055850 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.055821    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"]
Jul 07 18:07:41.058949 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.058927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.062249 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.062226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89337714-openshift-default-dockercfg-6f6d8\""
Jul 07 18:07:41.083930 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.083904    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"]
Jul 07 18:07:41.207353 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-socket\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207353 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-data\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207525 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-credential-socket\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207525 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-token\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207525 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istiod-ca-cert\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207623 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-certs\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207623 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-podinfo\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207623 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4br\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-kube-api-access-9x4br\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.207722 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.207634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-envoy\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.308785 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-socket\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.308785 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-data\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.308785 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-credential-socket\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-token\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istiod-ca-cert\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-certs\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-podinfo\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4br\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-kube-api-access-9x4br\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.308927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-envoy\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309455 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.309220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-data\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.309626 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.309604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-credential-socket\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.310787 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.310102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istiod-ca-cert\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.310787 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.310395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-certs\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.310787 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.310635    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-socket\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.311716 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.311690    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-envoy\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.312618 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.312592    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-podinfo\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.320001 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.319955    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-token\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.320885 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.320857    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4br\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-kube-api-access-9x4br\") pod \"e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") " pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.373733 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.373694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:41.399296 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.399271    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:41.511388 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-socket\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.511574 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511397    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-data\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.511711 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511659    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:07:41.511848 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9z88\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-kube-api-access-z9z88\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.511848 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-istio-token\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.511848 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511822    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-credential-socket\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.512021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-certs\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.512021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-envoy\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.512021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d711e9-593e-4801-b619-3041e1bb17d5-istio-podinfo\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.512021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.511989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d711e9-593e-4801-b619-3041e1bb17d5-istiod-ca-cert\") pod \"51d711e9-593e-4801-b619-3041e1bb17d5\" (UID: \"51d711e9-593e-4801-b619-3041e1bb17d5\") "
Jul 07 18:07:41.512687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.512241    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:07:41.512687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.512336    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.512687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.512353    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.512687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.512452    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:07:41.512687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.512540    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d711e9-593e-4801-b619-3041e1bb17d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:07:41.512687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.512661    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:07:41.514492 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.514462    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-kube-api-access-z9z88" (OuterVolumeSpecName: "kube-api-access-z9z88") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "kube-api-access-z9z88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:07:41.514694 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.514673    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"]
Jul 07 18:07:41.514758 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.514738    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:07:41.514916 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.514897    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d711e9-593e-4801-b619-3041e1bb17d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:07:41.515135 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.515096    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "51d711e9-593e-4801-b619-3041e1bb17d5" (UID: "51d711e9-593e-4801-b619-3041e1bb17d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:07:41.517948 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:07:41.517928    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c855acb_98f0_4695_867f_41ecc6ea2ffa.slice/crio-f0a83bba8c8960a045b47a963886a239dadb85c8cbe9d0a261b84e68cc54021a WatchSource:0}: Error finding container f0a83bba8c8960a045b47a963886a239dadb85c8cbe9d0a261b84e68cc54021a: Status 404 returned error can't find the container with id f0a83bba8c8960a045b47a963886a239dadb85c8cbe9d0a261b84e68cc54021a
Jul 07 18:07:41.520190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.520161    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:41.520279 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.520239    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:41.520339 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.520286    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:41.613803 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613760    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9z88\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-kube-api-access-z9z88\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.613803 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613806    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d711e9-593e-4801-b619-3041e1bb17d5-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.613987 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613822    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.613987 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613838    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.613987 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613851    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d711e9-593e-4801-b619-3041e1bb17d5-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.613987 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613863    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d711e9-593e-4801-b619-3041e1bb17d5-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:41.613987 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:41.613875    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d711e9-593e-4801-b619-3041e1bb17d5-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:07:42.242650 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.242617    2577 generic.go:356] "Generic (PLEG): container finished" podID="51d711e9-593e-4801-b619-3041e1bb17d5" containerID="3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627" exitCode=0
Jul 07 18:07:42.243070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.242671    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"
Jul 07 18:07:42.243070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.242709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg" event={"ID":"51d711e9-593e-4801-b619-3041e1bb17d5","Type":"ContainerDied","Data":"3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627"}
Jul 07 18:07:42.243070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.242767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg" event={"ID":"51d711e9-593e-4801-b619-3041e1bb17d5","Type":"ContainerDied","Data":"cc0b2aed9b914db0a148ab28c5ff454aea51cc9dc620a789f283b3f5cb3ed974"}
Jul 07 18:07:42.243070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.242788    2577 scope.go:117] "RemoveContainer" containerID="3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627"
Jul 07 18:07:42.244495 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.244467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" event={"ID":"7c855acb-98f0-4695-867f-41ecc6ea2ffa","Type":"ContainerStarted","Data":"096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"}
Jul 07 18:07:42.244594 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.244506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" event={"ID":"7c855acb-98f0-4695-867f-41ecc6ea2ffa","Type":"ContainerStarted","Data":"f0a83bba8c8960a045b47a963886a239dadb85c8cbe9d0a261b84e68cc54021a"}
Jul 07 18:07:42.251623 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.251608    2577 scope.go:117] "RemoveContainer" containerID="3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627"
Jul 07 18:07:42.251822 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:07:42.251806    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627\": container with ID starting with 3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627 not found: ID does not exist" containerID="3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627"
Jul 07 18:07:42.251886 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.251834    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627"} err="failed to get container status \"3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627\": rpc error: code = NotFound desc = could not find container \"3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627\": container with ID starting with 3661168dde619eabe4c8e48108fd043f30dee5561ca34aaf986027f4695cc627 not found: ID does not exist"
Jul 07 18:07:42.275484 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.275439    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" podStartSLOduration=1.275424211 podStartE2EDuration="1.275424211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:07:42.272378578 +0000 UTC m=+1623.846325918" watchObservedRunningTime="2026-07-07 18:07:42.275424211 +0000 UTC m=+1623.849371551"
Jul 07 18:07:42.310616 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.310588    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"]
Jul 07 18:07:42.321958 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.321934    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-afd9b7ef-openshift-default-86c49996ff-tzjkg"]
Jul 07 18:07:42.374897 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.374866    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:42.379820 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:42.379795    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:43.097355 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:43.097323    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51d711e9-593e-4801-b619-3041e1bb17d5" path="/var/lib/kubelet/pods/51d711e9-593e-4801-b619-3041e1bb17d5/volumes"
Jul 07 18:07:43.250540 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:43.250514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:43.251614 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:43.251595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:45.102084 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:45.102055    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:07:52.294046 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:52.294014    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerID="096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455" exitCode=0
Jul 07 18:07:52.294447 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:52.294058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" event={"ID":"7c855acb-98f0-4695-867f-41ecc6ea2ffa","Type":"ContainerDied","Data":"096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"}
Jul 07 18:07:52.294486 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:52.294462    2577 scope.go:117] "RemoveContainer" containerID="096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"
Jul 07 18:07:52.295465 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:52.295437    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:52.295536 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:52.295509    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:52.295584 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:52.295540    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:07:53.300657 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:53.300627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" event={"ID":"7c855acb-98f0-4695-867f-41ecc6ea2ffa","Type":"ContainerStarted","Data":"0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c"}
Jul 07 18:07:53.374762 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:53.374729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:53.378563 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:53.378537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:53.590133 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:53.590038    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"]
Jul 07 18:07:54.305594 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:54.305558    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:54.306040 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:54.305617    2577 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" secret="" err="secret \"e2e-ait-89337714-openshift-default-dockercfg-6f6d8\" not found"
Jul 07 18:07:54.306580 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:54.306564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:07:55.311006 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:55.310933    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy" containerID="cri-o://0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c" gracePeriod=30
Jul 07 18:07:55.810977 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:55.810942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:07:59.797056 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.797022    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"]
Jul 07 18:07:59.797552 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.797404    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d711e9-593e-4801-b619-3041e1bb17d5" containerName="istio-proxy"
Jul 07 18:07:59.797552 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.797417    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d711e9-593e-4801-b619-3041e1bb17d5" containerName="istio-proxy"
Jul 07 18:07:59.797552 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.797482    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d711e9-593e-4801-b619-3041e1bb17d5" containerName="istio-proxy"
Jul 07 18:07:59.799615 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.799595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.802295 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.802276    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-663072c9-openshift-default-dockercfg-mc7cl\""
Jul 07 18:07:59.813871 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.813847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"]
Jul 07 18:07:59.866239 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-data\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866391 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-envoy\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866391 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-socket\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866391 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-credential-socket\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866391 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzpz\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-kube-api-access-rqzpz\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866561 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-certs\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866561 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-istio-token\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866561 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe299e88-4776-4131-8572-e273500d35b5-istio-podinfo\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.866561 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.866463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe299e88-4776-4131-8572-e273500d35b5-istiod-ca-cert\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.966962 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.966922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-istio-token\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.966962 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.966961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe299e88-4776-4131-8572-e273500d35b5-istio-podinfo\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967258 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.966983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe299e88-4776-4131-8572-e273500d35b5-istiod-ca-cert\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967258 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-data\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967258 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-envoy\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967258 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-socket\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-credential-socket\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzpz\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-kube-api-access-rqzpz\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-certs\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967600 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-data\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967663 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-credential-socket\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967706 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967692    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-socket\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.967749 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.967725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-certs\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.968077 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.968062    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe299e88-4776-4131-8572-e273500d35b5-istiod-ca-cert\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.969372 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.969355    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-envoy\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.969457 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.969438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe299e88-4776-4131-8572-e273500d35b5-istio-podinfo\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.975456 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.975433    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzpz\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-kube-api-access-rqzpz\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:07:59.976826 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:07:59.976787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-istio-token\") pod \"e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") " pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:00.112462 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.112370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:00.263898 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.263872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"]
Jul 07 18:08:00.266186 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:08:00.266160    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe299e88_4776_4131_8572_e273500d35b5.slice/crio-ca7baa346a1c69422852383fa62a8510ca8c37e144993e87e3bc7605ea0be474 WatchSource:0}: Error finding container ca7baa346a1c69422852383fa62a8510ca8c37e144993e87e3bc7605ea0be474: Status 404 returned error can't find the container with id ca7baa346a1c69422852383fa62a8510ca8c37e144993e87e3bc7605ea0be474
Jul 07 18:08:00.268691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.268662    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:08:00.268793 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.268736    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:08:00.268793 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.268778    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:08:00.337275 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.337241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m" event={"ID":"fe299e88-4776-4131-8572-e273500d35b5","Type":"ContainerStarted","Data":"c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35"}
Jul 07 18:08:00.337385 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.337310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m" event={"ID":"fe299e88-4776-4131-8572-e273500d35b5","Type":"ContainerStarted","Data":"ca7baa346a1c69422852383fa62a8510ca8c37e144993e87e3bc7605ea0be474"}
Jul 07 18:08:00.360001 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.359942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m" podStartSLOduration=1.359926352 podStartE2EDuration="1.359926352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:08:00.358175707 +0000 UTC m=+1641.932123050" watchObservedRunningTime="2026-07-07 18:08:00.359926352 +0000 UTC m=+1641.933873673"
Jul 07 18:08:00.558806 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.558780    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:08:00.571632 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istiod-ca-cert\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.571814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-socket\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.571814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-data\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.571814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-certs\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.571814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-podinfo\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.571814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-credential-socket\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.572071 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-envoy\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.572071 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571874    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-token\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.572071 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571945    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x4br\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-kube-api-access-9x4br\") pod \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\" (UID: \"7c855acb-98f0-4695-867f-41ecc6ea2ffa\") "
Jul 07 18:08:00.572071 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.571951    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:00.572071 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572045    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:00.572348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572002    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:08:00.572348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572216    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.572348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572235    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.572348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572251    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.572745 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572711    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:00.572984 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.572958    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:00.574150 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.574102    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:00.574150 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.574120    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:08:00.574380 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.574360    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-kube-api-access-9x4br" (OuterVolumeSpecName: "kube-api-access-9x4br") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "kube-api-access-9x4br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:08:00.574446 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.574427    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c855acb-98f0-4695-867f-41ecc6ea2ffa" (UID: "7c855acb-98f0-4695-867f-41ecc6ea2ffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:08:00.672792 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.672759    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x4br\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-kube-api-access-9x4br\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.672792 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.672788    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.672792 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.672798    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.673016 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.672806    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.673016 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.672814    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:00.673016 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:00.672822    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c855acb-98f0-4695-867f-41ecc6ea2ffa-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:01.112510 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.112478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:01.117297 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.117276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:01.344510 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.344474    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerID="0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c" exitCode=0
Jul 07 18:08:01.344703 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.344539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"
Jul 07 18:08:01.344703 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.344562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" event={"ID":"7c855acb-98f0-4695-867f-41ecc6ea2ffa","Type":"ContainerDied","Data":"0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c"}
Jul 07 18:08:01.344703 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.344623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l" event={"ID":"7c855acb-98f0-4695-867f-41ecc6ea2ffa","Type":"ContainerDied","Data":"f0a83bba8c8960a045b47a963886a239dadb85c8cbe9d0a261b84e68cc54021a"}
Jul 07 18:08:01.344703 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.344645    2577 scope.go:117] "RemoveContainer" containerID="0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c"
Jul 07 18:08:01.345317 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.345297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:01.346180 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.346086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:01.353385 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.353366    2577 scope.go:117] "RemoveContainer" containerID="096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"
Jul 07 18:08:01.361137 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.361100    2577 scope.go:117] "RemoveContainer" containerID="0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c"
Jul 07 18:08:01.361417 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:08:01.361397    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c\": container with ID starting with 0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c not found: ID does not exist" containerID="0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c"
Jul 07 18:08:01.361480 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.361426    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c"} err="failed to get container status \"0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c\": rpc error: code = NotFound desc = could not find container \"0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c\": container with ID starting with 0a290513b5ba3fd847f4e429a3edff33b2ee24a066c28ff3b0909f09fbdf742c not found: ID does not exist"
Jul 07 18:08:01.361480 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.361446    2577 scope.go:117] "RemoveContainer" containerID="096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"
Jul 07 18:08:01.361683 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:08:01.361665    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455\": container with ID starting with 096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455 not found: ID does not exist" containerID="096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"
Jul 07 18:08:01.361741 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.361689    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455"} err="failed to get container status \"096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455\": rpc error: code = NotFound desc = could not find container \"096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455\": container with ID starting with 096f42ea0337c942e4a21a7492c56fd94719d8cb02948647df258472e76c4455 not found: ID does not exist"
Jul 07 18:08:01.369427 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.369401    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"]
Jul 07 18:08:01.384240 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:01.384212    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89337714-openshift-default-686bc4b8f9-f6x2l"]
Jul 07 18:08:03.096996 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:03.096961    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" path="/var/lib/kubelet/pods/7c855acb-98f0-4695-867f-41ecc6ea2ffa/volumes"
Jul 07 18:08:05.611102 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:05.611064    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"]
Jul 07 18:08:05.611945 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:05.611908    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m" podUID="fe299e88-4776-4131-8572-e273500d35b5" containerName="istio-proxy" containerID="cri-o://c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35" gracePeriod=30
Jul 07 18:08:06.616212 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:06.616181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:08:10.860950 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.860926    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:10.951173 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-istio-token\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951316 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-envoy\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951316 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe299e88-4776-4131-8572-e273500d35b5-istiod-ca-cert\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951316 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqzpz\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-kube-api-access-rqzpz\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951420 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-certs\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951470 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-socket\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951470 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951457    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-credential-socket\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951573 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe299e88-4776-4131-8572-e273500d35b5-istio-podinfo\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951573 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-data\") pod \"fe299e88-4776-4131-8572-e273500d35b5\" (UID: \"fe299e88-4776-4131-8572-e273500d35b5\") "
Jul 07 18:08:10.951676 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951620    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe299e88-4776-4131-8572-e273500d35b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:08:10.951745 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951677    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:10.951875 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951836    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe299e88-4776-4131-8572-e273500d35b5-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:10.951875 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951843    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:10.951875 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951853    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:10.951875 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951859    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:10.952031 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.951907    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:10.953404 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.953380    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-kube-api-access-rqzpz" (OuterVolumeSpecName: "kube-api-access-rqzpz") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "kube-api-access-rqzpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:08:10.953519 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.953502    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe299e88-4776-4131-8572-e273500d35b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:08:10.953623 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.953609    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:08:10.953678 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:10.953629    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe299e88-4776-4131-8572-e273500d35b5" (UID: "fe299e88-4776-4131-8572-e273500d35b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:08:11.052980 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.052945    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.052980 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.052974    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.052980 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.052986    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe299e88-4776-4131-8572-e273500d35b5-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.053211 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.052995    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.053211 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.053003    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.053211 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.053011    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe299e88-4776-4131-8572-e273500d35b5-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.053211 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.053018    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzpz\" (UniqueName: \"kubernetes.io/projected/fe299e88-4776-4131-8572-e273500d35b5-kube-api-access-rqzpz\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:08:11.393581 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.393543    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe299e88-4776-4131-8572-e273500d35b5" containerID="c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35" exitCode=0
Jul 07 18:08:11.393760 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.393591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m" event={"ID":"fe299e88-4776-4131-8572-e273500d35b5","Type":"ContainerDied","Data":"c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35"}
Jul 07 18:08:11.393760 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.393606    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"
Jul 07 18:08:11.393760 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.393657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m" event={"ID":"fe299e88-4776-4131-8572-e273500d35b5","Type":"ContainerDied","Data":"ca7baa346a1c69422852383fa62a8510ca8c37e144993e87e3bc7605ea0be474"}
Jul 07 18:08:11.393760 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.393678    2577 scope.go:117] "RemoveContainer" containerID="c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35"
Jul 07 18:08:11.402767 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.402747    2577 scope.go:117] "RemoveContainer" containerID="c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35"
Jul 07 18:08:11.403039 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:08:11.403021    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35\": container with ID starting with c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35 not found: ID does not exist" containerID="c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35"
Jul 07 18:08:11.403103 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.403045    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35"} err="failed to get container status \"c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35\": rpc error: code = NotFound desc = could not find container \"c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35\": container with ID starting with c42d74ff969adcb5295b1bb90c8a4cc8430e8d23e9f94e083246ff3c5e600d35 not found: ID does not exist"
Jul 07 18:08:11.415710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.415690    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"]
Jul 07 18:08:11.418232 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:11.418212    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-663072c9-openshift-default-64fc4dcfbc-src7m"]
Jul 07 18:08:12.616548 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:12.616513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:08:13.098438 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:08:13.098408    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe299e88-4776-4131-8572-e273500d35b5" path="/var/lib/kubelet/pods/fe299e88-4776-4131-8572-e273500d35b5/volumes"
Jul 07 18:09:34.323544 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:34.323507    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:09:41.939485 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:41.939449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:09:48.547740 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.547706    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"]
Jul 07 18:09:48.548244 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548225    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy"
Jul 07 18:09:48.548331 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548246    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy"
Jul 07 18:09:48.548331 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy"
Jul 07 18:09:48.548331 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548274    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy"
Jul 07 18:09:48.548331 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548283    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe299e88-4776-4131-8572-e273500d35b5" containerName="istio-proxy"
Jul 07 18:09:48.548331 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548292    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe299e88-4776-4131-8572-e273500d35b5" containerName="istio-proxy"
Jul 07 18:09:48.548555 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548386    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe299e88-4776-4131-8572-e273500d35b5" containerName="istio-proxy"
Jul 07 18:09:48.548555 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.548399    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy"
Jul 07 18:09:48.551400 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.551379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.554023 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.554006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5f113509-openshift-default-dockercfg-lx55n\""
Jul 07 18:09:48.578850 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.578826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"]
Jul 07 18:09:48.677574 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-socket\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677574 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-envoy\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677791 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgq5m\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-kube-api-access-cgq5m\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677791 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-podinfo\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677791 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istiod-ca-cert\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677791 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-credential-socket\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677791 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-data\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677961 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-token\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.677961 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.677858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-certs\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.778841 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.778808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-podinfo\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.778852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istiod-ca-cert\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.778901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-credential-socket\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.778934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-data\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779182 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-token\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779182 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-certs\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779277 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-socket\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779277 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-envoy\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779277 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgq5m\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-kube-api-access-cgq5m\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779503 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779457    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-credential-socket\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779627 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779579    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-certs\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779754 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istiod-ca-cert\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779849 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-socket\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.779967 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.779947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-data\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.781463 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.781443    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-podinfo\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.781554 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.781443    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-envoy\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.788069 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.788042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-token\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.788647 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.788629    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgq5m\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-kube-api-access-cgq5m\") pod \"e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") " pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:48.862299 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:48.862212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:49.008144 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.008120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"]
Jul 07 18:09:49.009609 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:09:49.009581    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0d42af_c9fc_48eb_8eb7_d1f480949311.slice/crio-6bf72e7b4cb89fcc246adad3da518eb69aff66b0c73f153c3b0741a68f45f89e WatchSource:0}: Error finding container 6bf72e7b4cb89fcc246adad3da518eb69aff66b0c73f153c3b0741a68f45f89e: Status 404 returned error can't find the container with id 6bf72e7b4cb89fcc246adad3da518eb69aff66b0c73f153c3b0741a68f45f89e
Jul 07 18:09:49.011770 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.011742    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:09:49.011828 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.011803    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:09:49.011866 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.011833    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:09:49.862995 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.862957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" event={"ID":"fb0d42af-c9fc-48eb-8eb7-d1f480949311","Type":"ContainerStarted","Data":"e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"}
Jul 07 18:09:49.863524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.863021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" event={"ID":"fb0d42af-c9fc-48eb-8eb7-d1f480949311","Type":"ContainerStarted","Data":"6bf72e7b4cb89fcc246adad3da518eb69aff66b0c73f153c3b0741a68f45f89e"}
Jul 07 18:09:49.888693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:49.888635    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" podStartSLOduration=1.8886195730000002 podStartE2EDuration="1.888619573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:09:49.887179332 +0000 UTC m=+1751.461126675" watchObservedRunningTime="2026-07-07 18:09:49.888619573 +0000 UTC m=+1751.462566914"
Jul 07 18:09:50.715336 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:50.715299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:09:50.862711 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:50.862680    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:50.868175 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:50.868149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:51.872412 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:51.872374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:09:51.873338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:09:51.873313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:00.903815 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:00.903779    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:10:04.934224 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:04.934189    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerID="e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc" exitCode=0
Jul 07 18:10:04.934656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:04.934262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" event={"ID":"fb0d42af-c9fc-48eb-8eb7-d1f480949311","Type":"ContainerDied","Data":"e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"}
Jul 07 18:10:04.934656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:04.934639    2577 scope.go:117] "RemoveContainer" containerID="e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"
Jul 07 18:10:04.935762 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:04.935736    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:04.935831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:04.935808    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:04.935866 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:04.935844    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:05.943834 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:05.943791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" event={"ID":"fb0d42af-c9fc-48eb-8eb7-d1f480949311","Type":"ContainerStarted","Data":"01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe"}
Jul 07 18:10:06.862758 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:06.862718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:06.863764 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:06.863734    2577 patch_prober.go:28] interesting pod/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 18:10:06.863879 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:06.863793    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 18:10:06.872680 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:06.872661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:07.863318 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:07.863283    2577 patch_prober.go:28] interesting pod/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 18:10:07.863717 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:07.863352    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 18:10:08.601318 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:08.601285    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:10:08.862641 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:08.862554    2577 patch_prober.go:28] interesting pod/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 07 18:10:08.862641 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:08.862615    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 07 18:10:09.866327 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:09.866294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:09.867086 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:09.867067    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:10.281926 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:10.281893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"]
Jul 07 18:10:10.972047 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:10.972005    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy" containerID="cri-o://01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe" gracePeriod=30
Jul 07 18:10:16.226735 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.226709    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:16.316729 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istiod-ca-cert\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.316901 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-socket\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.316901 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-certs\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.316901 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-credential-socket\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.316901 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-envoy\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.317089 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-podinfo\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.317089 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-data\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.317089 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.316952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-token\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.317089 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgq5m\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-kube-api-access-cgq5m\") pod \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\" (UID: \"fb0d42af-c9fc-48eb-8eb7-d1f480949311\") "
Jul 07 18:10:16.317333 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317155    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:10:16.317333 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317164    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:10:16.317333 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317273    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:10:16.317333 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317205    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:10:16.317527 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317353    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.317527 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317371    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.318016 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.317989    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:10:16.319193 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.319162    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-kube-api-access-cgq5m" (OuterVolumeSpecName: "kube-api-access-cgq5m") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "kube-api-access-cgq5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:10:16.319317 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.319280    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:10:16.319562 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.319538    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:10:16.319562 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.319546    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb0d42af-c9fc-48eb-8eb7-d1f480949311" (UID: "fb0d42af-c9fc-48eb-8eb7-d1f480949311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:10:16.417936 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417900    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgq5m\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-kube-api-access-cgq5m\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.417936 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417929    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.417936 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417940    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.418190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417949    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.418190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417958    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.418190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417965    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:16.418190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:16.417973    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0d42af-c9fc-48eb-8eb7-d1f480949311-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:10:17.000276 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.000236    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerID="01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe" exitCode=0
Jul 07 18:10:17.000448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.000281    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"
Jul 07 18:10:17.000448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.000301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" event={"ID":"fb0d42af-c9fc-48eb-8eb7-d1f480949311","Type":"ContainerDied","Data":"01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe"}
Jul 07 18:10:17.000448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.000351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl" event={"ID":"fb0d42af-c9fc-48eb-8eb7-d1f480949311","Type":"ContainerDied","Data":"6bf72e7b4cb89fcc246adad3da518eb69aff66b0c73f153c3b0741a68f45f89e"}
Jul 07 18:10:17.000448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.000374    2577 scope.go:117] "RemoveContainer" containerID="01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe"
Jul 07 18:10:17.009550 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.009532    2577 scope.go:117] "RemoveContainer" containerID="e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"
Jul 07 18:10:17.017166 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.017138    2577 scope.go:117] "RemoveContainer" containerID="01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe"
Jul 07 18:10:17.017391 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:10:17.017371    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe\": container with ID starting with 01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe not found: ID does not exist" containerID="01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe"
Jul 07 18:10:17.017439 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.017401    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe"} err="failed to get container status \"01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe\": rpc error: code = NotFound desc = could not find container \"01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe\": container with ID starting with 01aa2b045de8e1e7090942b07efcd8206502fc54706bc6ee847148fce8c762fe not found: ID does not exist"
Jul 07 18:10:17.017439 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.017419    2577 scope.go:117] "RemoveContainer" containerID="e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"
Jul 07 18:10:17.017629 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:10:17.017610    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc\": container with ID starting with e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc not found: ID does not exist" containerID="e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"
Jul 07 18:10:17.017668 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.017634    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc"} err="failed to get container status \"e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc\": rpc error: code = NotFound desc = could not find container \"e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc\": container with ID starting with e36d1bc6350f8a645ff03148c507c4c804b3db0cde28fe2bbfad2aa8f9349bdc not found: ID does not exist"
Jul 07 18:10:17.023761 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.023739    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"]
Jul 07 18:10:17.030468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.030448    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5f113509-openshift-default-5775f7c5d7-lbrtl"]
Jul 07 18:10:17.097755 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:17.097725    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" path="/var/lib/kubelet/pods/fb0d42af-c9fc-48eb-8eb7-d1f480949311/volumes"
Jul 07 18:10:30.027572 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027541    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"]
Jul 07 18:10:30.027925 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027880    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy"
Jul 07 18:10:30.027925 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027891    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy"
Jul 07 18:10:30.027925 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027919    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy"
Jul 07 18:10:30.028027 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027927    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy"
Jul 07 18:10:30.028027 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027978    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy"
Jul 07 18:10:30.028027 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.027990    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c855acb-98f0-4695-867f-41ecc6ea2ffa" containerName="istio-proxy"
Jul 07 18:10:30.032556 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.032535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.035005 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.034974    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8aa60534-openshift-default-dockercfg-6mjxb\""
Jul 07 18:10:30.047346 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.047323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"]
Jul 07 18:10:30.138503 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-certs\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138503 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-socket\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138503 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-envoy\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b122435-abef-4ea2-851c-94a344ba9316-istio-podinfo\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b122435-abef-4ea2-851c-94a344ba9316-istiod-ca-cert\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-istio-token\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-data\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.138777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-credential-socket\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.139095 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.138794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnndc\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-kube-api-access-tnndc\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.239949 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.239913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-envoy\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.239949 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.239950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b122435-abef-4ea2-851c-94a344ba9316-istio-podinfo\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.239976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b122435-abef-4ea2-851c-94a344ba9316-istiod-ca-cert\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-istio-token\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-data\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-credential-socket\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnndc\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-kube-api-access-tnndc\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-certs\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-socket\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240840 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240813    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-credential-socket\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240924 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240825    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b122435-abef-4ea2-851c-94a344ba9316-istiod-ca-cert\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240924 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240864    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-data\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.240924 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240894    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-certs\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.241032 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.240987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-socket\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.242318 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.242298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-envoy\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.242543 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.242522    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b122435-abef-4ea2-851c-94a344ba9316-istio-podinfo\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.260741 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.254937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-istio-token\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.260741 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.255364    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnndc\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-kube-api-access-tnndc\") pod \"e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") " pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.343315 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.343241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:30.480532 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.480503    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"]
Jul 07 18:10:30.481602 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:10:30.481580    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b122435_abef_4ea2_851c_94a344ba9316.slice/crio-119e0a517640955d033cdef296e70b1110aa1e7b28ed9a298e52e6be0589425d WatchSource:0}: Error finding container 119e0a517640955d033cdef296e70b1110aa1e7b28ed9a298e52e6be0589425d: Status 404 returned error can't find the container with id 119e0a517640955d033cdef296e70b1110aa1e7b28ed9a298e52e6be0589425d
Jul 07 18:10:30.483707 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.483672    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:30.483769 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.483728    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:30.483769 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:30.483760    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:31.070660 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:31.070623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" event={"ID":"6b122435-abef-4ea2-851c-94a344ba9316","Type":"ContainerStarted","Data":"da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"}
Jul 07 18:10:31.071121 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:31.070687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" event={"ID":"6b122435-abef-4ea2-851c-94a344ba9316","Type":"ContainerStarted","Data":"119e0a517640955d033cdef296e70b1110aa1e7b28ed9a298e52e6be0589425d"}
Jul 07 18:10:31.095834 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:31.095774    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" podStartSLOduration=1.095757804 podStartE2EDuration="1.095757804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:10:31.094267085 +0000 UTC m=+1792.668214426" watchObservedRunningTime="2026-07-07 18:10:31.095757804 +0000 UTC m=+1792.669705144"
Jul 07 18:10:31.344462 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:31.344379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:31.349826 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:31.349802    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:32.075855 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:32.075827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:32.076888 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:32.076871    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:37.810252 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:37.810215    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:10:38.998942 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:38.998916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:10:39.002824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:39.002803    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:10:46.139298 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:46.139264    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b122435-abef-4ea2-851c-94a344ba9316" containerID="da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892" exitCode=0
Jul 07 18:10:46.139740 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:46.139313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" event={"ID":"6b122435-abef-4ea2-851c-94a344ba9316","Type":"ContainerDied","Data":"da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"}
Jul 07 18:10:46.139740 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:46.139704    2577 scope.go:117] "RemoveContainer" containerID="da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"
Jul 07 18:10:46.140651 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:46.140624    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:46.140743 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:46.140688    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:46.140743 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:46.140722    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:10:47.076683 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:47.076647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:47.149358 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:47.149316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" event={"ID":"6b122435-abef-4ea2-851c-94a344ba9316","Type":"ContainerStarted","Data":"39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb"}
Jul 07 18:10:47.345638 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:47.345551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:47.345969 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:47.345943    2577 patch_prober.go:28] interesting pod/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 07 18:10:47.346139 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:47.346096    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 07 18:10:48.344407 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:48.344372    2577 patch_prober.go:28] interesting pod/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 07 18:10:48.345382 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:48.344965    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 07 18:10:49.343927 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:49.343881    2577 patch_prober.go:28] interesting pod/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 07 18:10:49.344263 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:49.343951    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 07 18:10:50.344654 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:50.344605    2577 patch_prober.go:28] interesting pod/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 07 18:10:50.345186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:50.344686    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 07 18:10:51.347780 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:51.347752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:51.348224 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:51.348037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:10:51.348749 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:10:51.348735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:11:04.897350 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.897312    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"]
Jul 07 18:11:04.897804 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.897790    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0d42af-c9fc-48eb-8eb7-d1f480949311" containerName="istio-proxy"
Jul 07 18:11:04.900278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.900263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:04.902844 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.902822    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8aa60534\"/\"openshift-service-ca.crt\""
Jul 07 18:11:04.903640 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.903620    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8aa60534\"/\"kube-root-ca.crt\""
Jul 07 18:11:04.903811 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.903792    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8aa60534\"/\"default-dockercfg-bpfdv\""
Jul 07 18:11:04.903879 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.903854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8aa60534\"/\"e2e-lifecycle-model-8aa60534-kserve-self-signed-certs\""
Jul 07 18:11:04.912948 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.912926    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"]
Jul 07 18:11:04.943011 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.942983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-home\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:04.943183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.943044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tls-certs\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:04.943183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.943084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tmp-dir\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:04.943183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.943101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-dshm\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:04.943183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.943174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-model-cache\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:04.943331 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:04.943210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtf28\" (UniqueName: \"kubernetes.io/projected/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-kube-api-access-qtf28\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044312 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tmp-dir\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044312 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-dshm\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044564 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-model-cache\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044564 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtf28\" (UniqueName: \"kubernetes.io/projected/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-kube-api-access-qtf28\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044564 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-home\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044564 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tls-certs\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.044782 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.044717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tmp-dir\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.045286 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.045257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-model-cache\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.045603 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.045583    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-home\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.047761 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.047741    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tls-certs\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.048347 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.048320    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-dshm\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.053436 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.053409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtf28\" (UniqueName: \"kubernetes.io/projected/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-kube-api-access-qtf28\") pod \"e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") " pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.210011 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.209906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:05.341868 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:05.341839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"]
Jul 07 18:11:05.343719 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:11:05.343689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fc31024_613b_43fa_a5cc_9dc4e7cbda35.slice/crio-34343e446722c62f37e84874e5687eb865f1d6d650fc5870eb5603d25d8e4699 WatchSource:0}: Error finding container 34343e446722c62f37e84874e5687eb865f1d6d650fc5870eb5603d25d8e4699: Status 404 returned error can't find the container with id 34343e446722c62f37e84874e5687eb865f1d6d650fc5870eb5603d25d8e4699
Jul 07 18:11:06.238549 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:06.238514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg" event={"ID":"0fc31024-613b-43fa-a5cc-9dc4e7cbda35","Type":"ContainerStarted","Data":"8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d"}
Jul 07 18:11:06.238549 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:06.238561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg" event={"ID":"0fc31024-613b-43fa-a5cc-9dc4e7cbda35","Type":"ContainerStarted","Data":"34343e446722c62f37e84874e5687eb865f1d6d650fc5870eb5603d25d8e4699"}
Jul 07 18:11:06.254958 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:06.254911    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg" podStartSLOduration=2.254896565 podStartE2EDuration="2.254896565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:11:06.254032813 +0000 UTC m=+1827.827980155" watchObservedRunningTime="2026-07-07 18:11:06.254896565 +0000 UTC m=+1827.828843905"
Jul 07 18:11:08.015512 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:08.015471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:11:15.210952 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:15.210899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:15.211426 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:15.211044    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:15.223558 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:15.223535    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:15.290956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:15.290922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:22.810047 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:22.810003    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:11:36.711301 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:36.711265    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"]
Jul 07 18:11:37.101559 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:37.101527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:11:51.731378 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:51.731336    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg" podUID="0fc31024-613b-43fa-a5cc-9dc4e7cbda35" containerName="main" containerID="cri-o://8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d" gracePeriod=45
Jul 07 18:11:51.983548 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:51.983493    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:52.054237 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054204    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tmp-dir\") pod \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") "
Jul 07 18:11:52.054420 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tls-certs\") pod \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") "
Jul 07 18:11:52.054420 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-dshm\") pod \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") "
Jul 07 18:11:52.054420 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtf28\" (UniqueName: \"kubernetes.io/projected/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-kube-api-access-qtf28\") pod \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") "
Jul 07 18:11:52.054420 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-model-cache\") pod \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") "
Jul 07 18:11:52.054621 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-home\") pod \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\" (UID: \"0fc31024-613b-43fa-a5cc-9dc4e7cbda35\") "
Jul 07 18:11:52.054621 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054482    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fc31024-613b-43fa-a5cc-9dc4e7cbda35" (UID: "0fc31024-613b-43fa-a5cc-9dc4e7cbda35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:11:52.054717 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054668    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fc31024-613b-43fa-a5cc-9dc4e7cbda35" (UID: "0fc31024-613b-43fa-a5cc-9dc4e7cbda35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:11:52.054717 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054703    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-home" (OuterVolumeSpecName: "home") pod "0fc31024-613b-43fa-a5cc-9dc4e7cbda35" (UID: "0fc31024-613b-43fa-a5cc-9dc4e7cbda35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:11:52.054798 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.054712    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:11:52.056460 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.056437    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fc31024-613b-43fa-a5cc-9dc4e7cbda35" (UID: "0fc31024-613b-43fa-a5cc-9dc4e7cbda35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:11:52.056553 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.056499    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-kube-api-access-qtf28" (OuterVolumeSpecName: "kube-api-access-qtf28") pod "0fc31024-613b-43fa-a5cc-9dc4e7cbda35" (UID: "0fc31024-613b-43fa-a5cc-9dc4e7cbda35"). InnerVolumeSpecName "kube-api-access-qtf28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:11:52.056553 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.056519    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-dshm" (OuterVolumeSpecName: "dshm") pod "0fc31024-613b-43fa-a5cc-9dc4e7cbda35" (UID: "0fc31024-613b-43fa-a5cc-9dc4e7cbda35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:11:52.155524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.155493    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:11:52.155524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.155518    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:11:52.155524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.155526    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtf28\" (UniqueName: \"kubernetes.io/projected/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-kube-api-access-qtf28\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:11:52.155752 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.155536    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:11:52.155752 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.155544    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc31024-613b-43fa-a5cc-9dc4e7cbda35-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:11:52.450800 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.450769    2577 generic.go:356] "Generic (PLEG): container finished" podID="0fc31024-613b-43fa-a5cc-9dc4e7cbda35" containerID="8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d" exitCode=0
Jul 07 18:11:52.450973 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.450812    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"
Jul 07 18:11:52.450973 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.450862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg" event={"ID":"0fc31024-613b-43fa-a5cc-9dc4e7cbda35","Type":"ContainerDied","Data":"8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d"}
Jul 07 18:11:52.450973 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.450907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg" event={"ID":"0fc31024-613b-43fa-a5cc-9dc4e7cbda35","Type":"ContainerDied","Data":"34343e446722c62f37e84874e5687eb865f1d6d650fc5870eb5603d25d8e4699"}
Jul 07 18:11:52.450973 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.450926    2577 scope.go:117] "RemoveContainer" containerID="8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d"
Jul 07 18:11:52.462289 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.462270    2577 scope.go:117] "RemoveContainer" containerID="8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d"
Jul 07 18:11:52.462538 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:11:52.462519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d\": container with ID starting with 8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d not found: ID does not exist" containerID="8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d"
Jul 07 18:11:52.462591 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.462548    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d"} err="failed to get container status \"8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d\": rpc error: code = NotFound desc = could not find container \"8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d\": container with ID starting with 8fbe7585cc8381b44fe142ff99a10e69f67c1c4628f83967ed1d1f510bed2b6d not found: ID does not exist"
Jul 07 18:11:52.474807 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.474785    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"]
Jul 07 18:11:52.479313 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:52.479291    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8aa60534/e2e-lifecycle-model-8aa60534-kserve-66cbbfdb99-9qgxg"]
Jul 07 18:11:53.096750 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:11:53.096713    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc31024-613b-43fa-a5cc-9dc4e7cbda35" path="/var/lib/kubelet/pods/0fc31024-613b-43fa-a5cc-9dc4e7cbda35/volumes"
Jul 07 18:12:26.812413 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:26.812233    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"]
Jul 07 18:12:26.812949 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:26.812499    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy" containerID="cri-o://39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb" gracePeriod=30
Jul 07 18:12:27.306119 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:27.306075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:12:32.072166 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.072088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:12:32.186747 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-credential-socket\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.186942 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnndc\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-kube-api-access-tnndc\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.186942 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b122435-abef-4ea2-851c-94a344ba9316-istio-podinfo\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.186942 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-data\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.186942 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-certs\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.187209 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-envoy\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.187209 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.186979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-socket\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.187209 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-istio-token\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.187209 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b122435-abef-4ea2-851c-94a344ba9316-istiod-ca-cert\") pod \"6b122435-abef-4ea2-851c-94a344ba9316\" (UID: \"6b122435-abef-4ea2-851c-94a344ba9316\") "
Jul 07 18:12:32.187209 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187074    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:12:32.187461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187228    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:12:32.187461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187255    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:12:32.187461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187370    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.187461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187391    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.187461 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187404    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.187669 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187517    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b122435-abef-4ea2-851c-94a344ba9316-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:12:32.187923 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.187894    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:12:32.189262 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.189232    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:12:32.189366 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.189327    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-kube-api-access-tnndc" (OuterVolumeSpecName: "kube-api-access-tnndc") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "kube-api-access-tnndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:12:32.189544 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.189516    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:12:32.189544 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.189513    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b122435-abef-4ea2-851c-94a344ba9316-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b122435-abef-4ea2-851c-94a344ba9316" (UID: "6b122435-abef-4ea2-851c-94a344ba9316"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:12:32.288271 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.288236    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnndc\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-kube-api-access-tnndc\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.288271 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.288265    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b122435-abef-4ea2-851c-94a344ba9316-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.288271 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.288274    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.288491 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.288282    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b122435-abef-4ea2-851c-94a344ba9316-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.288491 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.288291    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b122435-abef-4ea2-851c-94a344ba9316-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.288491 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.288298    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b122435-abef-4ea2-851c-94a344ba9316-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:12:32.635230 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.635197    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b122435-abef-4ea2-851c-94a344ba9316" containerID="39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb" exitCode=0
Jul 07 18:12:32.635416 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.635243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" event={"ID":"6b122435-abef-4ea2-851c-94a344ba9316","Type":"ContainerDied","Data":"39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb"}
Jul 07 18:12:32.635416 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.635281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2" event={"ID":"6b122435-abef-4ea2-851c-94a344ba9316","Type":"ContainerDied","Data":"119e0a517640955d033cdef296e70b1110aa1e7b28ed9a298e52e6be0589425d"}
Jul 07 18:12:32.635416 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.635301    2577 scope.go:117] "RemoveContainer" containerID="39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb"
Jul 07 18:12:32.635416 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.635246    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"
Jul 07 18:12:32.644360 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.644342    2577 scope.go:117] "RemoveContainer" containerID="da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"
Jul 07 18:12:32.651844 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.651827    2577 scope.go:117] "RemoveContainer" containerID="39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb"
Jul 07 18:12:32.652190 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:12:32.652090    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb\": container with ID starting with 39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb not found: ID does not exist" containerID="39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb"
Jul 07 18:12:32.652190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.652134    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb"} err="failed to get container status \"39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb\": rpc error: code = NotFound desc = could not find container \"39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb\": container with ID starting with 39f0b2db8950e4ca6f5d5ebfdd46353a4e24bc2a5b9b40de10548a78079036fb not found: ID does not exist"
Jul 07 18:12:32.652190 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.652154    2577 scope.go:117] "RemoveContainer" containerID="da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"
Jul 07 18:12:32.652399 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:12:32.652382    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892\": container with ID starting with da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892 not found: ID does not exist" containerID="da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"
Jul 07 18:12:32.652442 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.652405    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892"} err="failed to get container status \"da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892\": rpc error: code = NotFound desc = could not find container \"da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892\": container with ID starting with da7e7f562dbe8eecafd4881068dcb4c03dc71b4c7a4946607a1adb5ceccd6892 not found: ID does not exist"
Jul 07 18:12:32.655900 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.655882    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"]
Jul 07 18:12:32.659688 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:32.659665    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8aa60534-openshift-default-66bddb8d9f-24rj2"]
Jul 07 18:12:33.097818 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:12:33.097783    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b122435-abef-4ea2-851c-94a344ba9316" path="/var/lib/kubelet/pods/6b122435-abef-4ea2-851c-94a344ba9316/volumes"
Jul 07 18:13:22.256222 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256191    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"]
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256567    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256578    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256585    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256591    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256601    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc31024-613b-43fa-a5cc-9dc4e7cbda35" containerName="main"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256607    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc31024-613b-43fa-a5cc-9dc4e7cbda35" containerName="main"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256658    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc31024-613b-43fa-a5cc-9dc4e7cbda35" containerName="main"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256665    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy"
Jul 07 18:13:22.256691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.256672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b122435-abef-4ea2-851c-94a344ba9316" containerName="istio-proxy"
Jul 07 18:13:22.259688 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.259670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.262368 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.262350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-19c08a-openshift-default-dockercfg-rt7pq\""
Jul 07 18:13:22.274541 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.274522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"]
Jul 07 18:13:22.408293 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-socket\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408293 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istiod-ca-cert\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqzv\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-kube-api-access-zvqzv\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-data\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-token\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-podinfo\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-certs\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408663 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-credential-socket\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.408663 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.408568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-envoy\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509423 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-data\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509423 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-token\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509423 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-podinfo\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-certs\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-credential-socket\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-envoy\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-socket\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istiod-ca-cert\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509710 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqzv\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-kube-api-access-zvqzv\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509982 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509848    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-certs\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.509982 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.509966    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-data\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.510096 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.510075    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-socket\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.510459 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.510436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istiod-ca-cert\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.510558 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.510541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-credential-socket\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.511896 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.511876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-envoy\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.512028 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.512012    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-podinfo\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.517621 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.517596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqzv\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-kube-api-access-zvqzv\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.517717 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.517594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-token\") pod \"e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") " pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.570098 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.570063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:22.709323 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.709297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"]
Jul 07 18:13:22.711272 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:13:22.711245    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf52b53d_ae71_4e8c_bd57_d5dd3fdc81bc.slice/crio-4d8bcec036c7f45ac87a6529a6f8c5dc348588119e21cbb517b0a56a0b207251 WatchSource:0}: Error finding container 4d8bcec036c7f45ac87a6529a6f8c5dc348588119e21cbb517b0a56a0b207251: Status 404 returned error can't find the container with id 4d8bcec036c7f45ac87a6529a6f8c5dc348588119e21cbb517b0a56a0b207251
Jul 07 18:13:22.713792 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.713765    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:22.713859 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.713828    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:22.713901 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.713860    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:22.863312 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.863280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" event={"ID":"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc","Type":"ContainerStarted","Data":"b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"}
Jul 07 18:13:22.863478 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.863330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" event={"ID":"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc","Type":"ContainerStarted","Data":"4d8bcec036c7f45ac87a6529a6f8c5dc348588119e21cbb517b0a56a0b207251"}
Jul 07 18:13:22.884824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:22.884765    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" podStartSLOduration=0.884742112 podStartE2EDuration="884.742112ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:13:22.881935467 +0000 UTC m=+1964.455882809" watchObservedRunningTime="2026-07-07 18:13:22.884742112 +0000 UTC m=+1964.458689453"
Jul 07 18:13:23.570259 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:23.570224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:23.575370 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:23.575346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:23.868435 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:23.868355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:23.869389 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:23.869360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:29.103241 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:29.103209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:13:30.088723 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.088687    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"]
Jul 07 18:13:30.092871 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.092849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.100601 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.100577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ba4c79-openshift-default-dockercfg-tnh2k\""
Jul 07 18:13:30.103240 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.103216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"]
Jul 07 18:13:30.179855 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.179762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-credential-socket\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180470 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-certs\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180470 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bff015f-42d3-4941-999c-0299d6f924f7-istiod-ca-cert\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180788 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bff015f-42d3-4941-999c-0299d6f924f7-istio-podinfo\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180788 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-istio-token\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180788 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wfc\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-kube-api-access-84wfc\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180788 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-socket\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180990 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-data\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.180990 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.180847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-envoy\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.281763 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-data\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.281763 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-envoy\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-credential-socket\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-certs\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bff015f-42d3-4941-999c-0299d6f924f7-istiod-ca-cert\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bff015f-42d3-4941-999c-0299d6f924f7-istio-podinfo\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-istio-token\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.281958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wfc\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-kube-api-access-84wfc\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282036 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.282004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-socket\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282328 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.282198    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-data\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282328 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.282314    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-certs\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282571 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.282548    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-credential-socket\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282657 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.282585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-socket\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.282719 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.282662    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bff015f-42d3-4941-999c-0299d6f924f7-istiod-ca-cert\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.284468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.284450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-envoy\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.284607 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.284586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bff015f-42d3-4941-999c-0299d6f924f7-istio-podinfo\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.293836 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.293814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-istio-token\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.294386 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.294365    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wfc\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-kube-api-access-84wfc\") pod \"e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") " pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.412186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.412082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:30.605692 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.605664    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"]
Jul 07 18:13:30.608048 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:13:30.608016    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bff015f_42d3_4941_999c_0299d6f924f7.slice/crio-e1b1358e684a290d7962a80ad5cb7a00719e7a2d486d79062ef03d067a38d148 WatchSource:0}: Error finding container e1b1358e684a290d7962a80ad5cb7a00719e7a2d486d79062ef03d067a38d148: Status 404 returned error can't find the container with id e1b1358e684a290d7962a80ad5cb7a00719e7a2d486d79062ef03d067a38d148
Jul 07 18:13:30.610044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.610014    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:30.610125 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.610076    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:30.610170 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.610120    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:30.902850 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.902810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" event={"ID":"8bff015f-42d3-4941-999c-0299d6f924f7","Type":"ContainerStarted","Data":"4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"}
Jul 07 18:13:30.903061 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.902869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" event={"ID":"8bff015f-42d3-4941-999c-0299d6f924f7","Type":"ContainerStarted","Data":"e1b1358e684a290d7962a80ad5cb7a00719e7a2d486d79062ef03d067a38d148"}
Jul 07 18:13:30.922772 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:30.922696    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podStartSLOduration=0.922669466 podStartE2EDuration="922.669466ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:13:30.920617384 +0000 UTC m=+1972.494564722" watchObservedRunningTime="2026-07-07 18:13:30.922669466 +0000 UTC m=+1972.496616808"
Jul 07 18:13:31.412609 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:31.412568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:31.417444 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:31.417419    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:31.908411 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:31.908382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:31.909282 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:31.909266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:37.940004 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:37.939970    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerID="b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74" exitCode=0
Jul 07 18:13:37.940444 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:37.940037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" event={"ID":"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc","Type":"ContainerDied","Data":"b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"}
Jul 07 18:13:37.940444 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:37.940440    2577 scope.go:117] "RemoveContainer" containerID="b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"
Jul 07 18:13:37.941468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:37.941412    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:37.941529 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:37.941480    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:37.941529 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:37.941513    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:38.013963 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:38.013934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:13:38.868406 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:38.868369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:38.953422 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:38.952504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" event={"ID":"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc","Type":"ContainerStarted","Data":"2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563"}
Jul 07 18:13:39.571490 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:39.571456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:39.572430 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:39.572400    2577 patch_prober.go:28] interesting pod/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 18:13:39.572655 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:39.572629    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 18:13:40.570924 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:40.570873    2577 patch_prober.go:28] interesting pod/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 18:13:40.571482 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:40.570966    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 18:13:41.571013 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:41.570969    2577 patch_prober.go:28] interesting pod/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 18:13:41.571643 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:41.571602    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 18:13:42.571343 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:42.571303    2577 patch_prober.go:28] interesting pod/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 07 18:13:42.571906 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:42.571879    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 07 18:13:43.574574 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:43.574542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:43.574964 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:43.574946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:43.575587 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:43.575569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:13:46.910338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:46.910293    2577 patch_prober.go:28] interesting pod/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 18:13:46.910782 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:46.910350    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 18:13:47.002394 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:47.002357    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bff015f-42d3-4941-999c-0299d6f924f7" containerID="4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85" exitCode=0
Jul 07 18:13:47.002592 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:47.002430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" event={"ID":"8bff015f-42d3-4941-999c-0299d6f924f7","Type":"ContainerDied","Data":"4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"}
Jul 07 18:13:47.002904 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:47.002879    2577 scope.go:117] "RemoveContainer" containerID="4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"
Jul 07 18:13:47.003943 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:47.003915    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:47.004000 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:47.003982    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:47.004037 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:47.004011    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:13:48.008118 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:48.008085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" event={"ID":"8bff015f-42d3-4941-999c-0299d6f924f7","Type":"ContainerStarted","Data":"2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177"}
Jul 07 18:13:48.412899 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:48.412798    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:48.413172 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:48.413015    2577 patch_prober.go:28] interesting pod/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 18:13:48.413172 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:48.413070    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 18:13:49.412551 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:49.412517    2577 patch_prober.go:28] interesting pod/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 18:13:49.412959 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:49.412566    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 18:13:50.412647 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:50.412611    2577 patch_prober.go:28] interesting pod/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 18:13:50.413140 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:50.412674    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 18:13:51.412758 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:51.412720    2577 patch_prober.go:28] interesting pod/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 07 18:13:51.413171 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:51.412775    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 07 18:13:52.291506 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.291474    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"]
Jul 07 18:13:52.294225 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.294201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.297643 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.297621    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-19c08a\"/\"openshift-service-ca.crt\""
Jul 07 18:13:52.297785 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.297730    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-19c08a\"/\"default-dockercfg-pzpj8\""
Jul 07 18:13:52.297785 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.297758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-19c08a\"/\"test-model-19c08a-kserve-self-signed-certs\""
Jul 07 18:13:52.297785 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.297777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-19c08a\"/\"kube-root-ca.crt\""
Jul 07 18:13:52.303153 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.303129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"]
Jul 07 18:13:52.388219 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.388185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-tmp-dir\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.388219 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.388222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-home\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.388429 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.388249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-model-cache\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.388429 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.388340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-dshm\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.388429 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.388367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c81754-a4cd-4f52-8311-ce2559c346ab-tls-certs\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.388530 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.388448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfz8p\" (UniqueName: \"kubernetes.io/projected/a1c81754-a4cd-4f52-8311-ce2559c346ab-kube-api-access-mfz8p\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.417222 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.417186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:52.417592 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.417442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:52.418068 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.418053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:13:52.489657 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.489627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c81754-a4cd-4f52-8311-ce2559c346ab-tls-certs\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.489858 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.489733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfz8p\" (UniqueName: \"kubernetes.io/projected/a1c81754-a4cd-4f52-8311-ce2559c346ab-kube-api-access-mfz8p\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.489858 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.489780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-tmp-dir\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.489858 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.489829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-home\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.490029 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.489862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-model-cache\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.490029 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.489906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-dshm\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.490271 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.490250    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-home\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.490348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.490315    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-tmp-dir\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.490348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.490332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-model-cache\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.492465 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.492443    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-dshm\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.492680 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.492663    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c81754-a4cd-4f52-8311-ce2559c346ab-tls-certs\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.499030 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.498996    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfz8p\" (UniqueName: \"kubernetes.io/projected/a1c81754-a4cd-4f52-8311-ce2559c346ab-kube-api-access-mfz8p\") pod \"test-model-19c08a-kserve-7b5f457689-kx8rr\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") " pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.604355 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.604265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:13:52.733880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:52.733715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"]
Jul 07 18:13:52.736539 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:13:52.736513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c81754_a4cd_4f52_8311_ce2559c346ab.slice/crio-5ccf14fe41414ee6fd8ad4c6614eb4d598fad7b8538d22b393ce7e34a57fb2f2 WatchSource:0}: Error finding container 5ccf14fe41414ee6fd8ad4c6614eb4d598fad7b8538d22b393ce7e34a57fb2f2: Status 404 returned error can't find the container with id 5ccf14fe41414ee6fd8ad4c6614eb4d598fad7b8538d22b393ce7e34a57fb2f2
Jul 07 18:13:53.035632 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:53.035592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr" event={"ID":"a1c81754-a4cd-4f52-8311-ce2559c346ab","Type":"ContainerStarted","Data":"9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a"}
Jul 07 18:13:53.035825 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:53.035656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr" event={"ID":"a1c81754-a4cd-4f52-8311-ce2559c346ab","Type":"ContainerStarted","Data":"5ccf14fe41414ee6fd8ad4c6614eb4d598fad7b8538d22b393ce7e34a57fb2f2"}
Jul 07 18:13:53.051733 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:53.051684    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr" podStartSLOduration=1.051668547 podStartE2EDuration="1.051668547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:13:53.050968844 +0000 UTC m=+1994.624916198" watchObservedRunningTime="2026-07-07 18:13:53.051668547 +0000 UTC m=+1994.625615887"
Jul 07 18:13:55.999428 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:13:55.999391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:14:02.604533 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:02.604491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:14:02.604533 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:02.604542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:14:02.617027 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:02.617000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:14:03.097788 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:03.097756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:14:08.787680 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.787648    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"]
Jul 07 18:14:08.790553 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.790529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.793905 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.793881    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba4c79\"/\"default-dockercfg-5ssw5\""
Jul 07 18:14:08.794013 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.793909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba4c79\"/\"openshift-service-ca.crt\""
Jul 07 18:14:08.794013 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.793983    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ba4c79\"/\"test-model-ba4c79-kserve-self-signed-certs\""
Jul 07 18:14:08.794126 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.794022    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ba4c79\"/\"kube-root-ca.crt\""
Jul 07 18:14:08.802035 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.802015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"]
Jul 07 18:14:08.833908 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.833885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-dshm\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.834041 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.833922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-home\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.834041 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.833966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqpz\" (UniqueName: \"kubernetes.io/projected/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-kube-api-access-cwqpz\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.834200 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.834043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tls-certs\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.834200 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.834082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-model-cache\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.834200 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.834178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tmp-dir\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.934646 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.934612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tls-certs\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.934646 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.934648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-model-cache\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.934890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.934678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tmp-dir\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.934890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.934696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-dshm\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.934890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.934733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-home\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.934890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.934795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqpz\" (UniqueName: \"kubernetes.io/projected/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-kube-api-access-cwqpz\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.935169 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.935096    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-model-cache\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.935169 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.935135    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tmp-dir\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.935366 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.935252    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-home\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.937138 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.937095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-dshm\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.937350 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.937334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tls-certs\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:08.944630 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:08.944586    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqpz\" (UniqueName: \"kubernetes.io/projected/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-kube-api-access-cwqpz\") pod \"test-model-ba4c79-kserve-578b974b57-d7v28\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") " pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:09.101413 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:09.101335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:09.234288 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:09.234255    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"]
Jul 07 18:14:09.236265 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:14:09.236234    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5ba4820_edc0_4d86_9d9d_e99fa6cebf11.slice/crio-b01b3e187e512ba331e4e57532bf09b7d44f44cb77ce0dfe46fdeb779815be6c WatchSource:0}: Error finding container b01b3e187e512ba331e4e57532bf09b7d44f44cb77ce0dfe46fdeb779815be6c: Status 404 returned error can't find the container with id b01b3e187e512ba331e4e57532bf09b7d44f44cb77ce0dfe46fdeb779815be6c
Jul 07 18:14:10.116799 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:10.116763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28" event={"ID":"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11","Type":"ContainerStarted","Data":"12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726"}
Jul 07 18:14:10.117267 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:10.116814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28" event={"ID":"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11","Type":"ContainerStarted","Data":"b01b3e187e512ba331e4e57532bf09b7d44f44cb77ce0dfe46fdeb779815be6c"}
Jul 07 18:14:10.135478 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:10.135425    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28" podStartSLOduration=2.135401448 podStartE2EDuration="2.135401448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:14:10.133337717 +0000 UTC m=+2011.707285082" watchObservedRunningTime="2026-07-07 18:14:10.135401448 +0000 UTC m=+2011.709348790"
Jul 07 18:14:15.107173 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:15.107135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:14:19.101452 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:19.101424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:19.101921 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:19.101465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:19.113989 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:19.113965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:19.172597 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:19.172571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:30.917730 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:30.917696    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"]
Jul 07 18:14:31.904781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:31.904740    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"]
Jul 07 18:14:36.695567 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:36.695529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:14:45.938944 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:45.938896    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr" podUID="a1c81754-a4cd-4f52-8311-ce2559c346ab" containerName="main" containerID="cri-o://9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a" gracePeriod=45
Jul 07 18:14:46.191101 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.191040    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:14:46.268867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.268832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c81754-a4cd-4f52-8311-ce2559c346ab-tls-certs\") pod \"a1c81754-a4cd-4f52-8311-ce2559c346ab\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") "
Jul 07 18:14:46.268867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.268869    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-model-cache\") pod \"a1c81754-a4cd-4f52-8311-ce2559c346ab\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") "
Jul 07 18:14:46.269141 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.268891    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-tmp-dir\") pod \"a1c81754-a4cd-4f52-8311-ce2559c346ab\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") "
Jul 07 18:14:46.269141 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.268934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfz8p\" (UniqueName: \"kubernetes.io/projected/a1c81754-a4cd-4f52-8311-ce2559c346ab-kube-api-access-mfz8p\") pod \"a1c81754-a4cd-4f52-8311-ce2559c346ab\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") "
Jul 07 18:14:46.269141 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.268964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-home\") pod \"a1c81754-a4cd-4f52-8311-ce2559c346ab\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") "
Jul 07 18:14:46.269141 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.268993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-dshm\") pod \"a1c81754-a4cd-4f52-8311-ce2559c346ab\" (UID: \"a1c81754-a4cd-4f52-8311-ce2559c346ab\") "
Jul 07 18:14:46.269349 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.269172    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1c81754-a4cd-4f52-8311-ce2559c346ab" (UID: "a1c81754-a4cd-4f52-8311-ce2559c346ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:46.269349 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.269198    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1c81754-a4cd-4f52-8311-ce2559c346ab" (UID: "a1c81754-a4cd-4f52-8311-ce2559c346ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:46.269349 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.269334    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-home" (OuterVolumeSpecName: "home") pod "a1c81754-a4cd-4f52-8311-ce2559c346ab" (UID: "a1c81754-a4cd-4f52-8311-ce2559c346ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:46.269467 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.269344    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:46.269467 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.269392    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:46.271139 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.271092    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-dshm" (OuterVolumeSpecName: "dshm") pod "a1c81754-a4cd-4f52-8311-ce2559c346ab" (UID: "a1c81754-a4cd-4f52-8311-ce2559c346ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:46.271243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.271152    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c81754-a4cd-4f52-8311-ce2559c346ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1c81754-a4cd-4f52-8311-ce2559c346ab" (UID: "a1c81754-a4cd-4f52-8311-ce2559c346ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:14:46.271243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.271188    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c81754-a4cd-4f52-8311-ce2559c346ab-kube-api-access-mfz8p" (OuterVolumeSpecName: "kube-api-access-mfz8p") pod "a1c81754-a4cd-4f52-8311-ce2559c346ab" (UID: "a1c81754-a4cd-4f52-8311-ce2559c346ab"). InnerVolumeSpecName "kube-api-access-mfz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:14:46.289600 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.289572    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1c81754-a4cd-4f52-8311-ce2559c346ab" containerID="9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a" exitCode=0
Jul 07 18:14:46.289712 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.289613    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"
Jul 07 18:14:46.289712 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.289657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr" event={"ID":"a1c81754-a4cd-4f52-8311-ce2559c346ab","Type":"ContainerDied","Data":"9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a"}
Jul 07 18:14:46.289712 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.289697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr" event={"ID":"a1c81754-a4cd-4f52-8311-ce2559c346ab","Type":"ContainerDied","Data":"5ccf14fe41414ee6fd8ad4c6614eb4d598fad7b8538d22b393ce7e34a57fb2f2"}
Jul 07 18:14:46.289830 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.289715    2577 scope.go:117] "RemoveContainer" containerID="9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a"
Jul 07 18:14:46.298186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.298170    2577 scope.go:117] "RemoveContainer" containerID="9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a"
Jul 07 18:14:46.298453 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:14:46.298435    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a\": container with ID starting with 9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a not found: ID does not exist" containerID="9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a"
Jul 07 18:14:46.298494 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.298462    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a"} err="failed to get container status \"9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a\": rpc error: code = NotFound desc = could not find container \"9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a\": container with ID starting with 9c4b0384bca88976ed6b4a2ef7e014796d48f6ff35a29419aa30a60f2b3ffe9a not found: ID does not exist"
Jul 07 18:14:46.310761 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.310739    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"]
Jul 07 18:14:46.314301 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.314280    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-19c08a/test-model-19c08a-kserve-7b5f457689-kx8rr"]
Jul 07 18:14:46.369747 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.369720    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:46.369747 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.369745    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c81754-a4cd-4f52-8311-ce2559c346ab-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:46.369747 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.369754    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfz8p\" (UniqueName: \"kubernetes.io/projected/a1c81754-a4cd-4f52-8311-ce2559c346ab-kube-api-access-mfz8p\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:46.369950 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.369763    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c81754-a4cd-4f52-8311-ce2559c346ab-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:46.925138 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:46.925072    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28" podUID="f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" containerName="main" containerID="cri-o://12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726" gracePeriod=45
Jul 07 18:14:47.098966 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.098933    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c81754-a4cd-4f52-8311-ce2559c346ab" path="/var/lib/kubelet/pods/a1c81754-a4cd-4f52-8311-ce2559c346ab/volumes"
Jul 07 18:14:47.179990 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.179932    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:47.276687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tmp-dir\") pod \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") "
Jul 07 18:14:47.276849 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-home\") pod \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") "
Jul 07 18:14:47.276849 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tls-certs\") pod \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") "
Jul 07 18:14:47.276849 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwqpz\" (UniqueName: \"kubernetes.io/projected/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-kube-api-access-cwqpz\") pod \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") "
Jul 07 18:14:47.277015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-dshm\") pod \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") "
Jul 07 18:14:47.277015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-model-cache\") pod \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\" (UID: \"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11\") "
Jul 07 18:14:47.277015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276973    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-home" (OuterVolumeSpecName: "home") pod "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" (UID: "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:47.277015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.276980    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" (UID: "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:47.277295 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.277275    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" (UID: "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:47.277348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.277285    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:47.277348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.277318    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:47.278951 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.278926    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" (UID: "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:14:47.279067 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.278975    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-kube-api-access-cwqpz" (OuterVolumeSpecName: "kube-api-access-cwqpz") pod "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" (UID: "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11"). InnerVolumeSpecName "kube-api-access-cwqpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:14:47.279067 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.279026    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-dshm" (OuterVolumeSpecName: "dshm") pod "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" (UID: "f5ba4820-edc0-4d86-9d9d-e99fa6cebf11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:14:47.295334 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.295306    2577 generic.go:356] "Generic (PLEG): container finished" podID="f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" containerID="12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726" exitCode=0
Jul 07 18:14:47.295435 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.295340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28" event={"ID":"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11","Type":"ContainerDied","Data":"12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726"}
Jul 07 18:14:47.295435 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.295353    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"
Jul 07 18:14:47.295435 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.295382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28" event={"ID":"f5ba4820-edc0-4d86-9d9d-e99fa6cebf11","Type":"ContainerDied","Data":"b01b3e187e512ba331e4e57532bf09b7d44f44cb77ce0dfe46fdeb779815be6c"}
Jul 07 18:14:47.295435 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.295402    2577 scope.go:117] "RemoveContainer" containerID="12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726"
Jul 07 18:14:47.304475 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.304460    2577 scope.go:117] "RemoveContainer" containerID="12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726"
Jul 07 18:14:47.304696 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:14:47.304679    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726\": container with ID starting with 12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726 not found: ID does not exist" containerID="12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726"
Jul 07 18:14:47.304750 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.304705    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726"} err="failed to get container status \"12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726\": rpc error: code = NotFound desc = could not find container \"12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726\": container with ID starting with 12ad01c170f3aa9fe94a0bdbf6ff82a5d7e4a8358e4cf46ccbebd932198f8726 not found: ID does not exist"
Jul 07 18:14:47.316455 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.316432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"]
Jul 07 18:14:47.321202 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.321182    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ba4c79/test-model-ba4c79-kserve-578b974b57-d7v28"]
Jul 07 18:14:47.377795 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.377763    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:47.377795 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.377790    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:47.377795 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.377799    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:47.378027 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:47.377807    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqpz\" (UniqueName: \"kubernetes.io/projected/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11-kube-api-access-cwqpz\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:14:49.098093 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:14:49.098059    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" path="/var/lib/kubelet/pods/f5ba4820-edc0-4d86-9d9d-e99fa6cebf11/volumes"
Jul 07 18:15:14.607864 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:14.607772    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"]
Jul 07 18:15:14.608475 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:14.608018    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy" containerID="cri-o://2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563" gracePeriod=30
Jul 07 18:15:18.804473 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:18.804429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:15:19.857670 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.857648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:15:19.959341 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959249    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-certs\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959341 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-socket\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959341 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959326    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-token\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-podinfo\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istiod-ca-cert\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959568    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:19.959745 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-credential-socket\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959745 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-envoy\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959745 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959684    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:19.959745 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-data\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959940 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvqzv\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-kube-api-access-zvqzv\") pod \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\" (UID: \"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc\") "
Jul 07 18:15:19.959940 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959815    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:19.959940 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.959919    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:15:19.960078 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.960052    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:19.960144 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.960080    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:19.960144 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.960095    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:19.960144 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.960123    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:19.960733 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.960704    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:19.961781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.961756    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:15:19.961880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.961777    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:15:19.961880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.961802    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-kube-api-access-zvqzv" (OuterVolumeSpecName: "kube-api-access-zvqzv") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "kube-api-access-zvqzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:15:19.962007 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:19.961927    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" (UID: "bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:20.061342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.061296    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:20.061342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.061340    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqzv\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-kube-api-access-zvqzv\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:20.061342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.061354    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:20.061587 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.061366    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:20.061587 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.061376    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:20.468728 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.468696    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerID="2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563" exitCode=0
Jul 07 18:15:20.468920 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.468757    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"
Jul 07 18:15:20.468920 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.468763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" event={"ID":"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc","Type":"ContainerDied","Data":"2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563"}
Jul 07 18:15:20.468920 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.468894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n" event={"ID":"bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc","Type":"ContainerDied","Data":"4d8bcec036c7f45ac87a6529a6f8c5dc348588119e21cbb517b0a56a0b207251"}
Jul 07 18:15:20.468920 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.468915    2577 scope.go:117] "RemoveContainer" containerID="2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563"
Jul 07 18:15:20.477881 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.477865    2577 scope.go:117] "RemoveContainer" containerID="b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"
Jul 07 18:15:20.485332 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.485315    2577 scope.go:117] "RemoveContainer" containerID="2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563"
Jul 07 18:15:20.485578 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:15:20.485559    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563\": container with ID starting with 2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563 not found: ID does not exist" containerID="2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563"
Jul 07 18:15:20.485657 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.485590    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563"} err="failed to get container status \"2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563\": rpc error: code = NotFound desc = could not find container \"2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563\": container with ID starting with 2fd0b725af18edc1fa456991c0a62f6b44d0c4f94c301c1997ff014e98fe7563 not found: ID does not exist"
Jul 07 18:15:20.485657 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.485614    2577 scope.go:117] "RemoveContainer" containerID="b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"
Jul 07 18:15:20.485877 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:15:20.485859    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74\": container with ID starting with b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74 not found: ID does not exist" containerID="b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"
Jul 07 18:15:20.485916 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.485882    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74"} err="failed to get container status \"b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74\": rpc error: code = NotFound desc = could not find container \"b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74\": container with ID starting with b818bc0ae4fb6fe518419b92fe064db006706d52ff88c428166afb0a39663f74 not found: ID does not exist"
Jul 07 18:15:20.489264 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.489234    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"]
Jul 07 18:15:20.496215 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:20.496194    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-19c08a-openshift-default-86d49cdb7c-w284n"]
Jul 07 18:15:21.097089 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:21.097053    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" path="/var/lib/kubelet/pods/bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc/volumes"
Jul 07 18:15:21.433466 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:21.433376    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"]
Jul 07 18:15:21.433630 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:21.433607    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" containerID="cri-o://2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177" gracePeriod=30
Jul 07 18:15:22.385228 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385197    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"]
Jul 07 18:15:22.385569 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385527    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy"
Jul 07 18:15:22.385569 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385537    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy"
Jul 07 18:15:22.385569 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c81754-a4cd-4f52-8311-ce2559c346ab" containerName="main"
Jul 07 18:15:22.385569 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385554    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c81754-a4cd-4f52-8311-ce2559c346ab" containerName="main"
Jul 07 18:15:22.385569 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385564    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" containerName="main"
Jul 07 18:15:22.385569 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385570    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" containerName="main"
Jul 07 18:15:22.385768 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385625    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy"
Jul 07 18:15:22.385768 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385633    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy"
Jul 07 18:15:22.385768 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385641    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c81754-a4cd-4f52-8311-ce2559c346ab" containerName="main"
Jul 07 18:15:22.385768 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.385647    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ba4820-edc0-4d86-9d9d-e99fa6cebf11" containerName="main"
Jul 07 18:15:22.389735 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.389715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.392342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.392313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e88d61-openshift-default-dockercfg-ws78z\""
Jul 07 18:15:22.401810 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.401790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"]
Jul 07 18:15:22.419370 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.419343    2577 patch_prober.go:28] interesting pod/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 18:15:22.419490 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.419405    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 18:15:22.482468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9msd\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-kube-api-access-s9msd\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-data\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-envoy\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-token\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-podinfo\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istiod-ca-cert\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-socket\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482974 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-credential-socket\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.482974 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.482918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-certs\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.583990 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.583953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-podinfo\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.583990 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.583995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istiod-ca-cert\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584257 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-socket\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584257 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-credential-socket\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584257 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-certs\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584257 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9msd\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-kube-api-access-s9msd\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584466 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-data\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584466 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-envoy\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584466 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-token\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584609 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-credential-socket\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584609 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584572    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-socket\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-data\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584790    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istiod-ca-cert\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.584965 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.584880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-certs\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.586772 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.586745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-envoy\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.586992 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.586971    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-podinfo\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.592750 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.592724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9msd\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-kube-api-access-s9msd\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.592838 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.592771    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-token\") pod \"e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") " pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.700131 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.700033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:22.839853 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.839826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"]
Jul 07 18:15:22.842160 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:15:22.842131    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9663917a_f449_43d7_9ee9_3cf9869fe5b1.slice/crio-cd51411a9722ed4b39e1bc653452dfdf907ea0aca21a8ea453ebe83a214593e7 WatchSource:0}: Error finding container cd51411a9722ed4b39e1bc653452dfdf907ea0aca21a8ea453ebe83a214593e7: Status 404 returned error can't find the container with id cd51411a9722ed4b39e1bc653452dfdf907ea0aca21a8ea453ebe83a214593e7
Jul 07 18:15:22.844301 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.844270    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:22.844378 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.844352    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:22.844417 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:22.844377    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:23.486980 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:23.486936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" event={"ID":"9663917a-f449-43d7-9ee9-3cf9869fe5b1","Type":"ContainerStarted","Data":"11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"}
Jul 07 18:15:23.487424 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:23.486993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" event={"ID":"9663917a-f449-43d7-9ee9-3cf9869fe5b1","Type":"ContainerStarted","Data":"cd51411a9722ed4b39e1bc653452dfdf907ea0aca21a8ea453ebe83a214593e7"}
Jul 07 18:15:23.506956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:23.506912    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podStartSLOduration=1.506896748 podStartE2EDuration="1.506896748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:15:23.506474988 +0000 UTC m=+2085.080422331" watchObservedRunningTime="2026-07-07 18:15:23.506896748 +0000 UTC m=+2085.080844083"
Jul 07 18:15:23.700949 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:23.700909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:23.705852 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:23.705824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:24.491384 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:24.491348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:24.492346 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:24.492324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:26.692989 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.692924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:15:26.824662 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-socket\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.824867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-envoy\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.824867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bff015f-42d3-4941-999c-0299d6f924f7-istiod-ca-cert\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.824867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-credential-socket\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.824867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824777    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84wfc\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-kube-api-access-84wfc\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.824867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bff015f-42d3-4941-999c-0299d6f924f7-istio-podinfo\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.824867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-certs\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.825195 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824886    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-data\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.825195 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-istio-token\") pod \"8bff015f-42d3-4941-999c-0299d6f924f7\" (UID: \"8bff015f-42d3-4941-999c-0299d6f924f7\") "
Jul 07 18:15:26.825195 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.824993    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:26.825195 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.825044    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:26.825383 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.825284    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:26.825383 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.825290    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.825383 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.825332    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.825383 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.825331    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bff015f-42d3-4941-999c-0299d6f924f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:15:26.825801 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.825776    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:26.826970 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.826946    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-kube-api-access-84wfc" (OuterVolumeSpecName: "kube-api-access-84wfc") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "kube-api-access-84wfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:15:26.827201 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.827179    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bff015f-42d3-4941-999c-0299d6f924f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:15:26.827368 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.827343    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:26.827368 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.827357    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bff015f-42d3-4941-999c-0299d6f924f7" (UID: "8bff015f-42d3-4941-999c-0299d6f924f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:15:26.926278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926237    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bff015f-42d3-4941-999c-0299d6f924f7-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.926278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926272    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.926278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926286    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.926524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926296    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.926524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926307    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bff015f-42d3-4941-999c-0299d6f924f7-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.926524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926319    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bff015f-42d3-4941-999c-0299d6f924f7-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:26.926524 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:26.926330    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wfc\" (UniqueName: \"kubernetes.io/projected/8bff015f-42d3-4941-999c-0299d6f924f7-kube-api-access-84wfc\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:27.506999 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.506958    2577 generic.go:356] "Generic (PLEG): container finished" podID="8bff015f-42d3-4941-999c-0299d6f924f7" containerID="2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177" exitCode=0
Jul 07 18:15:27.507210 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.507019    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"
Jul 07 18:15:27.507210 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.507042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" event={"ID":"8bff015f-42d3-4941-999c-0299d6f924f7","Type":"ContainerDied","Data":"2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177"}
Jul 07 18:15:27.507210 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.507100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx" event={"ID":"8bff015f-42d3-4941-999c-0299d6f924f7","Type":"ContainerDied","Data":"e1b1358e684a290d7962a80ad5cb7a00719e7a2d486d79062ef03d067a38d148"}
Jul 07 18:15:27.507210 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.507146    2577 scope.go:117] "RemoveContainer" containerID="2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177"
Jul 07 18:15:27.515831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.515813    2577 scope.go:117] "RemoveContainer" containerID="4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"
Jul 07 18:15:27.523817 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.523800    2577 scope.go:117] "RemoveContainer" containerID="2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177"
Jul 07 18:15:27.524134 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:15:27.524093    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177\": container with ID starting with 2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177 not found: ID does not exist" containerID="2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177"
Jul 07 18:15:27.524225 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.524145    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177"} err="failed to get container status \"2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177\": rpc error: code = NotFound desc = could not find container \"2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177\": container with ID starting with 2bd4d8df9961f8debb902cfaa7970768d44d0a736baf9ea0b5c8c406eb199177 not found: ID does not exist"
Jul 07 18:15:27.524225 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.524167    2577 scope.go:117] "RemoveContainer" containerID="4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"
Jul 07 18:15:27.524355 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.524333    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"]
Jul 07 18:15:27.524446 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:15:27.524431    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85\": container with ID starting with 4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85 not found: ID does not exist" containerID="4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"
Jul 07 18:15:27.524481 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.524452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85"} err="failed to get container status \"4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85\": rpc error: code = NotFound desc = could not find container \"4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85\": container with ID starting with 4adcc39147a85bf54efaaf6f21c304bdee1e44a6ef7de31caa00c8a9f0becd85 not found: ID does not exist"
Jul 07 18:15:27.529438 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:27.529418    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ba4c79-openshift-default-8646d98747-ckhgx"]
Jul 07 18:15:29.097816 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:29.097785    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" path="/var/lib/kubelet/pods/8bff015f-42d3-4941-999c-0299d6f924f7/volumes"
Jul 07 18:15:30.783004 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.782959    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"]
Jul 07 18:15:30.783498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783405    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy"
Jul 07 18:15:30.783498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy"
Jul 07 18:15:30.783498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783434    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy"
Jul 07 18:15:30.783498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783440    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy"
Jul 07 18:15:30.783498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783446    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy"
Jul 07 18:15:30.783498 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf52b53d-ae71-4e8c-bd57-d5dd3fdc81bc" containerName="istio-proxy"
Jul 07 18:15:30.783842 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy"
Jul 07 18:15:30.783842 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.783516    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bff015f-42d3-4941-999c-0299d6f924f7" containerName="istio-proxy"
Jul 07 18:15:30.788118 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.788078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.790480 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.790459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9c804d-openshift-default-dockercfg-l2tw2\""
Jul 07 18:15:30.802065 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.802037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"]
Jul 07 18:15:30.961298 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-data\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961298 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjb6b\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-kube-api-access-bjb6b\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961540 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-certs\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961540 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95936c8e-b164-407f-bf33-27995304d4af-istiod-ca-cert\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961540 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-envoy\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961540 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-istio-token\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961743 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-credential-socket\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961743 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-socket\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:30.961743 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:30.961660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95936c8e-b164-407f-bf33-27995304d4af-istio-podinfo\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063176 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-certs\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063176 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95936c8e-b164-407f-bf33-27995304d4af-istiod-ca-cert\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063385 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-envoy\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063385 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-istio-token\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063385 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-credential-socket\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063541 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-socket\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063541 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95936c8e-b164-407f-bf33-27995304d4af-istio-podinfo\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063541 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063498    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-certs\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063541 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-data\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063760 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjb6b\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-kube-api-access-bjb6b\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063760 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-credential-socket\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063861 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063774    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-socket\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063917 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-data\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.063917 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.063867    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95936c8e-b164-407f-bf33-27995304d4af-istiod-ca-cert\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.065438 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.065415    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-envoy\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.065735 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.065718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95936c8e-b164-407f-bf33-27995304d4af-istio-podinfo\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.071392 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.071371    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-istio-token\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.071525 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.071507    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjb6b\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-kube-api-access-bjb6b\") pod \"e2e-api-b-9c804d-openshift-default-649495b985-4hwtp\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") " pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.106876 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.106852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:31.275999 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.275971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"]
Jul 07 18:15:31.277329 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:15:31.277305    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95936c8e_b164_407f_bf33_27995304d4af.slice/crio-a7b42cbde9e8ea80eca9a3d58b5e1e9d2e9a16216dc9ffccd4702405a92e8d56 WatchSource:0}: Error finding container a7b42cbde9e8ea80eca9a3d58b5e1e9d2e9a16216dc9ffccd4702405a92e8d56: Status 404 returned error can't find the container with id a7b42cbde9e8ea80eca9a3d58b5e1e9d2e9a16216dc9ffccd4702405a92e8d56
Jul 07 18:15:31.279408 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.279370    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:31.279491 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.279444    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:31.279491 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.279478    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:31.528686 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.528655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" event={"ID":"95936c8e-b164-407f-bf33-27995304d4af","Type":"ContainerStarted","Data":"9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"}
Jul 07 18:15:31.528854 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.528705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" event={"ID":"95936c8e-b164-407f-bf33-27995304d4af","Type":"ContainerStarted","Data":"a7b42cbde9e8ea80eca9a3d58b5e1e9d2e9a16216dc9ffccd4702405a92e8d56"}
Jul 07 18:15:31.546990 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:31.546937    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podStartSLOduration=1.546916618 podStartE2EDuration="1.546916618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:15:31.546230847 +0000 UTC m=+2093.120178211" watchObservedRunningTime="2026-07-07 18:15:31.546916618 +0000 UTC m=+2093.120863963"
Jul 07 18:15:32.107592 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:32.107555    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:32.112523 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:32.112499    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:32.535951 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:32.535917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:32.536815 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:32.536795    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:33.623303 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:33.623260    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:15:39.025867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:39.025842    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:15:39.029895 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:39.029873    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:15:40.533311 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:40.533269    2577 patch_prober.go:28] interesting pod/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 07 18:15:40.533688 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:40.533335    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 07 18:15:42.587957 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:42.587922    2577 generic.go:356] "Generic (PLEG): container finished" podID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerID="11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531" exitCode=0
Jul 07 18:15:42.588437 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:42.587995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" event={"ID":"9663917a-f449-43d7-9ee9-3cf9869fe5b1","Type":"ContainerDied","Data":"11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"}
Jul 07 18:15:42.588508 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:42.588492    2577 scope.go:117] "RemoveContainer" containerID="11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"
Jul 07 18:15:42.589596 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:42.589569    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:42.589667 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:42.589652    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:42.589708 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:42.589691    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:43.594019 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:43.593983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" event={"ID":"9663917a-f449-43d7-9ee9-3cf9869fe5b1","Type":"ContainerStarted","Data":"65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60"}
Jul 07 18:15:43.700961 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:43.700926    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:43.701176 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:43.701155    2577 patch_prober.go:28] interesting pod/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 07 18:15:43.701227 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:43.701203    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 07 18:15:44.700802 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:44.700767    2577 patch_prober.go:28] interesting pod/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 07 18:15:44.701247 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:44.700817    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 07 18:15:45.701441 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:45.701398    2577 patch_prober.go:28] interesting pod/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 07 18:15:45.701857 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:45.701475    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 07 18:15:46.700422 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:46.700381    2577 patch_prober.go:28] interesting pod/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 07 18:15:46.700699 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:46.700451    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 07 18:15:47.704598 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:47.704563    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:47.705145 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:47.704843    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:47.705560 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:47.705544    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:48.578301 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:48.578261    2577 patch_prober.go:28] interesting pod/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 07 18:15:48.578519 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:48.578325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 07 18:15:49.810854 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:49.810821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:15:50.628458 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:50.628424    2577 generic.go:356] "Generic (PLEG): container finished" podID="95936c8e-b164-407f-bf33-27995304d4af" containerID="9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4" exitCode=0
Jul 07 18:15:50.628707 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:50.628490    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" event={"ID":"95936c8e-b164-407f-bf33-27995304d4af","Type":"ContainerDied","Data":"9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"}
Jul 07 18:15:50.628913 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:50.628895    2577 scope.go:117] "RemoveContainer" containerID="9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"
Jul 07 18:15:50.629904 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:50.629880    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:50.629955 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:50.629945    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:50.629992 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:50.629978    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:15:51.638976 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:51.638935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" event={"ID":"95936c8e-b164-407f-bf33-27995304d4af","Type":"ContainerStarted","Data":"5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01"}
Jul 07 18:15:52.107289 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:52.107253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:52.107468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:52.107436    2577 patch_prober.go:28] interesting pod/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 07 18:15:52.107517 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:52.107485    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 07 18:15:53.011008 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:53.010969    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"]
Jul 07 18:15:53.011430 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:53.011202    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy" containerID="cri-o://65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60" gracePeriod=30
Jul 07 18:15:53.107447 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:53.107411    2577 patch_prober.go:28] interesting pod/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 07 18:15:53.107656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:53.107487    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 07 18:15:54.107737 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:54.107704    2577 patch_prober.go:28] interesting pod/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 07 18:15:54.108158 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:54.107758    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 07 18:15:55.107464 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:55.107428    2577 patch_prober.go:28] interesting pod/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 07 18:15:55.107647 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:55.107501    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 07 18:15:56.111686 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:56.111656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:56.112081 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:56.111899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:56.112661 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:56.112643    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:15:58.272748 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.272720    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:58.321535 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321452    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-podinfo\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321535 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321502    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istiod-ca-cert\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321742 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-socket\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321742 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9msd\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-kube-api-access-s9msd\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321742 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-envoy\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321742 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-credential-socket\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321742 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-data\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.321742 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-token\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.322056 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.321745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-certs\") pod \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\" (UID: \"9663917a-f449-43d7-9ee9-3cf9869fe5b1\") "
Jul 07 18:15:58.322056 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.322000    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:15:58.322278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.322236    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:58.322278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.322252    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:58.322407 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.322322    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:58.322817 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.322791    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:58.323890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.323860    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:15:58.323992 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.323906    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:15:58.324269 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.324250    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-kube-api-access-s9msd" (OuterVolumeSpecName: "kube-api-access-s9msd") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "kube-api-access-s9msd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:15:58.324325 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.324301    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "9663917a-f449-43d7-9ee9-3cf9869fe5b1" (UID: "9663917a-f449-43d7-9ee9-3cf9869fe5b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:15:58.422751 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422716    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422751 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422746    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9msd\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-kube-api-access-s9msd\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422751 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422756    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422765    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422774    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422782    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422789    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9663917a-f449-43d7-9ee9-3cf9869fe5b1-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422796    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.422981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.422803    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9663917a-f449-43d7-9ee9-3cf9869fe5b1-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:15:58.683656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.683575    2577 generic.go:356] "Generic (PLEG): container finished" podID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerID="65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60" exitCode=0
Jul 07 18:15:58.683656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.683622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"
Jul 07 18:15:58.683656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.683640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" event={"ID":"9663917a-f449-43d7-9ee9-3cf9869fe5b1","Type":"ContainerDied","Data":"65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60"}
Jul 07 18:15:58.683872 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.683704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v" event={"ID":"9663917a-f449-43d7-9ee9-3cf9869fe5b1","Type":"ContainerDied","Data":"cd51411a9722ed4b39e1bc653452dfdf907ea0aca21a8ea453ebe83a214593e7"}
Jul 07 18:15:58.683872 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.683726    2577 scope.go:117] "RemoveContainer" containerID="65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60"
Jul 07 18:15:58.693046 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.693029    2577 scope.go:117] "RemoveContainer" containerID="11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"
Jul 07 18:15:58.700667 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.700648    2577 scope.go:117] "RemoveContainer" containerID="65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60"
Jul 07 18:15:58.700909 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:15:58.700893    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60\": container with ID starting with 65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60 not found: ID does not exist" containerID="65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60"
Jul 07 18:15:58.700978 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.700915    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60"} err="failed to get container status \"65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60\": rpc error: code = NotFound desc = could not find container \"65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60\": container with ID starting with 65a78064f726f3812a1ffe1872a83fa1183279059138fa092a910d39b2f62d60 not found: ID does not exist"
Jul 07 18:15:58.700978 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.700932    2577 scope.go:117] "RemoveContainer" containerID="11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"
Jul 07 18:15:58.701192 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:15:58.701174    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531\": container with ID starting with 11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531 not found: ID does not exist" containerID="11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"
Jul 07 18:15:58.701238 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.701199    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531"} err="failed to get container status \"11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531\": rpc error: code = NotFound desc = could not find container \"11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531\": container with ID starting with 11fcc6feb366cef44e2065aae0f94a56498d150268f177f99979322c0910e531 not found: ID does not exist"
Jul 07 18:15:58.705493 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.705473    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"]
Jul 07 18:15:58.709854 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:58.709834    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e88d61-openshift-default-66bb5d59c9-z2t7v"]
Jul 07 18:15:59.097260 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:15:59.097226    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" path="/var/lib/kubelet/pods/9663917a-f449-43d7-9ee9-3cf9869fe5b1/volumes"
Jul 07 18:16:00.036347 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.036318    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"]
Jul 07 18:16:00.036777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.036497    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy" containerID="cri-o://5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01" gracePeriod=30
Jul 07 18:16:00.939849 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.939817    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"]
Jul 07 18:16:00.940285 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.940258    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy"
Jul 07 18:16:00.940285 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.940282    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy"
Jul 07 18:16:00.940483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.940306    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy"
Jul 07 18:16:00.940483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.940317    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy"
Jul 07 18:16:00.940483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.940391    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy"
Jul 07 18:16:00.940483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.940405    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9663917a-f449-43d7-9ee9-3cf9869fe5b1" containerName="istio-proxy"
Jul 07 18:16:00.943736 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.943720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:00.946185 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.946165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a68ed4-openshift-default-dockercfg-4w4jj\""
Jul 07 18:16:00.954451 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:00.954428    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"]
Jul 07 18:16:01.047226 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-certs\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ddc40-752d-4246-bc71-f1eb441d19e1-istiod-ca-cert\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-credential-socket\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-token\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-podinfo\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-data\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzv2\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-kube-api-access-ngzv2\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-socket\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.047608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.047484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-envoy\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148184 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzv2\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-kube-api-access-ngzv2\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148362 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-socket\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148362 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-envoy\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148362 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-certs\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148362 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ddc40-752d-4246-bc71-f1eb441d19e1-istiod-ca-cert\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148362 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-credential-socket\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148633 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-token\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148633 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-podinfo\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148633 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-data\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-certs\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148777 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-socket\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.148985 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.148959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-data\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.149062 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.149021    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-credential-socket\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.149263 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.149243    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ddc40-752d-4246-bc71-f1eb441d19e1-istiod-ca-cert\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.150556 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.150538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-envoy\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.150902 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.150885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-podinfo\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.157399 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.157371    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-token\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.157828 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.157803    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzv2\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-kube-api-access-ngzv2\") pod \"e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") " pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.255509 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.255475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:01.391728 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.391698    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"]
Jul 07 18:16:01.394382 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:16:01.394346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47ddc40_752d_4246_bc71_f1eb441d19e1.slice/crio-241cd57a4a57495f3e60a7db6e656690767bc5a24422c2c1bb5c86593258dda7 WatchSource:0}: Error finding container 241cd57a4a57495f3e60a7db6e656690767bc5a24422c2c1bb5c86593258dda7: Status 404 returned error can't find the container with id 241cd57a4a57495f3e60a7db6e656690767bc5a24422c2c1bb5c86593258dda7
Jul 07 18:16:01.396450 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.396412    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:01.396566 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.396498    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:01.396566 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.396548    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:01.706890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.706783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" event={"ID":"b47ddc40-752d-4246-bc71-f1eb441d19e1","Type":"ContainerStarted","Data":"2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"}
Jul 07 18:16:01.706890 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.706880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" event={"ID":"b47ddc40-752d-4246-bc71-f1eb441d19e1","Type":"ContainerStarted","Data":"241cd57a4a57495f3e60a7db6e656690767bc5a24422c2c1bb5c86593258dda7"}
Jul 07 18:16:01.725723 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:01.725670    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podStartSLOduration=1.725656264 podStartE2EDuration="1.725656264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:16:01.725137811 +0000 UTC m=+2123.299085147" watchObservedRunningTime="2026-07-07 18:16:01.725656264 +0000 UTC m=+2123.299603618"
Jul 07 18:16:02.256361 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:02.256320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:02.261744 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:02.261716    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:02.712070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:02.711982    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:02.712951 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:02.712927    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:02.897118 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:02.897074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:16:05.294283 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.294259    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:16:05.386797 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-data\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.386797 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95936c8e-b164-407f-bf33-27995304d4af-istiod-ca-cert\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386803    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-certs\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386819    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-socket\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjb6b\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-kube-api-access-bjb6b\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386873    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-envoy\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.386923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95936c8e-b164-407f-bf33-27995304d4af-istio-podinfo\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387001    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-istio-token\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387044 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-credential-socket\") pod \"95936c8e-b164-407f-bf33-27995304d4af\" (UID: \"95936c8e-b164-407f-bf33-27995304d4af\") "
Jul 07 18:16:05.387411 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387128    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:16:05.387411 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387170    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:16:05.387411 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387364    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:16:05.387582 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387425    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.387582 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387443    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.387582 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387567    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-data" (OuterVolumeSpecName: "istio-data") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:16:05.387736 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.387579    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95936c8e-b164-407f-bf33-27995304d4af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:16:05.389263 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.389228    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95936c8e-b164-407f-bf33-27995304d4af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:16:05.389263 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.389238    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:16:05.389263 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.389233    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-istio-token" (OuterVolumeSpecName: "istio-token") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:16:05.389263 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.389250    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-kube-api-access-bjb6b" (OuterVolumeSpecName: "kube-api-access-bjb6b") pod "95936c8e-b164-407f-bf33-27995304d4af" (UID: "95936c8e-b164-407f-bf33-27995304d4af"). InnerVolumeSpecName "kube-api-access-bjb6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:16:05.488449 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488416    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjb6b\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-kube-api-access-bjb6b\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.488449 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488444    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.488645 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488459    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95936c8e-b164-407f-bf33-27995304d4af-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.488645 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488468    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95936c8e-b164-407f-bf33-27995304d4af-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.488645 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488477    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.488645 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488488    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95936c8e-b164-407f-bf33-27995304d4af-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.488645 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.488497    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95936c8e-b164-407f-bf33-27995304d4af-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:16:05.726171 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.726068    2577 generic.go:356] "Generic (PLEG): container finished" podID="95936c8e-b164-407f-bf33-27995304d4af" containerID="5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01" exitCode=0
Jul 07 18:16:05.726171 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.726124    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"
Jul 07 18:16:05.726171 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.726151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" event={"ID":"95936c8e-b164-407f-bf33-27995304d4af","Type":"ContainerDied","Data":"5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01"}
Jul 07 18:16:05.726428 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.726201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp" event={"ID":"95936c8e-b164-407f-bf33-27995304d4af","Type":"ContainerDied","Data":"a7b42cbde9e8ea80eca9a3d58b5e1e9d2e9a16216dc9ffccd4702405a92e8d56"}
Jul 07 18:16:05.726428 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.726222    2577 scope.go:117] "RemoveContainer" containerID="5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01"
Jul 07 18:16:05.735483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.735467    2577 scope.go:117] "RemoveContainer" containerID="9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"
Jul 07 18:16:05.743047 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.743029    2577 scope.go:117] "RemoveContainer" containerID="5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01"
Jul 07 18:16:05.743318 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:16:05.743297    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01\": container with ID starting with 5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01 not found: ID does not exist" containerID="5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01"
Jul 07 18:16:05.743411 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.743325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01"} err="failed to get container status \"5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01\": rpc error: code = NotFound desc = could not find container \"5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01\": container with ID starting with 5b2abb2fc1045bcde24985e1c7f358e92449299c4cf417d2fcdb7b7e106ddb01 not found: ID does not exist"
Jul 07 18:16:05.743411 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.743343    2577 scope.go:117] "RemoveContainer" containerID="9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"
Jul 07 18:16:05.743555 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:16:05.743540    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4\": container with ID starting with 9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4 not found: ID does not exist" containerID="9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"
Jul 07 18:16:05.743594 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.743558    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4"} err="failed to get container status \"9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4\": rpc error: code = NotFound desc = could not find container \"9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4\": container with ID starting with 9923c5006a1812e3d32813a29312921ecd94ed8d791d8096d2025b4eadaea0f4 not found: ID does not exist"
Jul 07 18:16:05.746907 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.746885    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"]
Jul 07 18:16:05.751147 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:05.751128    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9c804d-openshift-default-649495b985-4hwtp"]
Jul 07 18:16:07.097120 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:07.097076    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95936c8e-b164-407f-bf33-27995304d4af" path="/var/lib/kubelet/pods/95936c8e-b164-407f-bf33-27995304d4af/volumes"
Jul 07 18:16:08.841608 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.841571    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"]
Jul 07 18:16:08.841984 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.841922    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy"
Jul 07 18:16:08.841984 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.841933    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy"
Jul 07 18:16:08.841984 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.841945    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy"
Jul 07 18:16:08.841984 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.841951    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy"
Jul 07 18:16:08.842158 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.842003    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy"
Jul 07 18:16:08.844893 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.844868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.847275 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.847246    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bc5a1a-openshift-default-dockercfg-48d4s\""
Jul 07 18:16:08.858260 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.858235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"]
Jul 07 18:16:08.916974 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.916931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-podinfo\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917162 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.916992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxlz\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-kube-api-access-6hxlz\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917162 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-envoy\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917162 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-socket\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917162 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-data\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-certs\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-token\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istiod-ca-cert\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:08.917342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:08.917293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-credential-socket\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.017786 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-socket\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.017956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-data\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.017956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-certs\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.017956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-token\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.017956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istiod-ca-cert\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.017956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-credential-socket\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018250 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.017990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-podinfo\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018250 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxlz\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-kube-api-access-6hxlz\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018250 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-envoy\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018250 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018230    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-socket\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018438 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018397    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-data\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018515 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-certs\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.018576 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-credential-socket\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.019015 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.018986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istiod-ca-cert\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.020370 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.020348    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-envoy\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.020607 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.020585    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-podinfo\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.025272 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.025249    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-token\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.025767 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.025748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxlz\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-kube-api-access-6hxlz\") pod \"e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") " pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.156328 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.156239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:09.308912 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.308880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"]
Jul 07 18:16:09.311066 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:16:09.311038    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6172ed2a_a67e_49bb_8bed_f60bec1b1ea5.slice/crio-9b95fbbb2b397e42764cf368c68632a6e5f35704d197b4fcd36cf9faf2fe3467 WatchSource:0}: Error finding container 9b95fbbb2b397e42764cf368c68632a6e5f35704d197b4fcd36cf9faf2fe3467: Status 404 returned error can't find the container with id 9b95fbbb2b397e42764cf368c68632a6e5f35704d197b4fcd36cf9faf2fe3467
Jul 07 18:16:09.313262 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.313233    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:09.313336 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.313298    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:09.313336 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.313329    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:09.747091 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.747050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" event={"ID":"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5","Type":"ContainerStarted","Data":"79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"}
Jul 07 18:16:09.747316 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.747128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" event={"ID":"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5","Type":"ContainerStarted","Data":"9b95fbbb2b397e42764cf368c68632a6e5f35704d197b4fcd36cf9faf2fe3467"}
Jul 07 18:16:09.769293 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:09.769238    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podStartSLOduration=1.769217203 podStartE2EDuration="1.769217203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:16:09.767068019 +0000 UTC m=+2131.341015361" watchObservedRunningTime="2026-07-07 18:16:09.769217203 +0000 UTC m=+2131.343164546"
Jul 07 18:16:10.157158 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:10.157078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:10.162265 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:10.162234    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:10.755424 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:10.755387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:10.756809 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:10.756704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:13.313952 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:13.313914    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:16:17.713399 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.713365    2577 patch_prober.go:28] interesting pod/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 07 18:16:17.713814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.713419    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 07 18:16:17.790593 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.790562    2577 generic.go:356] "Generic (PLEG): container finished" podID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerID="2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025" exitCode=0
Jul 07 18:16:17.790769 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.790635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" event={"ID":"b47ddc40-752d-4246-bc71-f1eb441d19e1","Type":"ContainerDied","Data":"2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"}
Jul 07 18:16:17.791033 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.791019    2577 scope.go:117] "RemoveContainer" containerID="2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"
Jul 07 18:16:17.792002 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.791973    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:17.792155 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.792053    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:17.792155 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:17.792103    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:18.797763 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:18.797725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" event={"ID":"b47ddc40-752d-4246-bc71-f1eb441d19e1","Type":"ContainerStarted","Data":"7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23"}
Jul 07 18:16:19.256634 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:19.256597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:19.256906 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:19.256876    2577 patch_prober.go:28] interesting pod/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 07 18:16:19.256992 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:19.256939    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 07 18:16:20.256822 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:20.256779    2577 patch_prober.go:28] interesting pod/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 07 18:16:20.257448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:20.257417    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 07 18:16:21.255914 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:21.255866    2577 patch_prober.go:28] interesting pod/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 07 18:16:21.256216 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:21.255935    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 07 18:16:22.258133 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:22.257938    2577 patch_prober.go:28] interesting pod/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 07 18:16:22.258133 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:22.258007    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 07 18:16:22.410513 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:22.410408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:16:23.260412 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:23.260382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:23.260914 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:23.260844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:23.261630 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:23.261610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:16:25.757239 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.757206    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 18:16:25.757677 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.757260    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 18:16:25.837030 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.837001    2577 generic.go:356] "Generic (PLEG): container finished" podID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerID="79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9" exitCode=0
Jul 07 18:16:25.837217 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.837051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" event={"ID":"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5","Type":"ContainerDied","Data":"79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"}
Jul 07 18:16:25.837463 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.837442    2577 scope.go:117] "RemoveContainer" containerID="79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"
Jul 07 18:16:25.838448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.838420    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:25.838549 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.838484    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:25.838549 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:25.838532    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 18:16:26.843656 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:26.843625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" event={"ID":"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5","Type":"ContainerStarted","Data":"7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336"}
Jul 07 18:16:27.157071 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:27.156959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:27.157277 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:27.157145    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 18:16:27.157277 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:27.157211    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 18:16:28.157161 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:28.157123    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 18:16:28.157564 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:28.157187    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 18:16:29.157599 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:29.157553    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 18:16:29.158133 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:29.157624    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 18:16:30.157163 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:30.157051    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 18:16:30.157163 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:30.157137    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 18:16:31.157451 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.157412    2577 patch_prober.go:28] interesting pod/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 07 18:16:31.157969 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.157466    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 07 18:16:31.890945 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.890910    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"]
Jul 07 18:16:31.891365 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.891350    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95936c8e-b164-407f-bf33-27995304d4af" containerName="istio-proxy"
Jul 07 18:16:31.893468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.893446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:31.896054 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.896031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a68ed4\"/\"openshift-service-ca.crt\""
Jul 07 18:16:31.896189 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.896038    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a68ed4\"/\"kube-root-ca.crt\""
Jul 07 18:16:31.896898 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.896878    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a68ed4\"/\"default-dockercfg-6htc5\""
Jul 07 18:16:31.897092 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.897079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a68ed4\"/\"auth-test-model-a68ed4-kserve-self-signed-certs\""
Jul 07 18:16:31.905847 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.905827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"]
Jul 07 18:16:31.960939 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.960908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tmp-dir\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:31.960939 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.960941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-model-cache\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:31.961178 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.960969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-home\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:31.961178 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.961033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd9j\" (UniqueName: \"kubernetes.io/projected/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-kube-api-access-dmd9j\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:31.961178 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.961088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-dshm\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:31.961178 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:31.961105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tls-certs\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.062604 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.062557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tmp-dir\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.062796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.062613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-model-cache\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.062796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.062662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-home\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.062796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.062700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd9j\" (UniqueName: \"kubernetes.io/projected/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-kube-api-access-dmd9j\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.062796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.062769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-dshm\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.062796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.062791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tls-certs\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.063086 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.063023    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tmp-dir\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.063345 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.063320    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-home\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.064399 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.063732    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-model-cache\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.066214 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.066188    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tls-certs\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.067021 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.066976    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-dshm\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.071634 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.071595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd9j\" (UniqueName: \"kubernetes.io/projected/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-kube-api-access-dmd9j\") pod \"auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") " pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.160824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.160744    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:32.161196 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.161000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:32.161614 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.161599    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:16:32.203631 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.203599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:32.343672 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.343647    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"]
Jul 07 18:16:32.346307 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:16:32.346281    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb71b991_68f0_4548_929d_2cf2a3bdaa9f.slice/crio-af74ad97c835adfed8b848d5e799e6bc7e8959a8504c01d72f5e27a8e187fa53 WatchSource:0}: Error finding container af74ad97c835adfed8b848d5e799e6bc7e8959a8504c01d72f5e27a8e187fa53: Status 404 returned error can't find the container with id af74ad97c835adfed8b848d5e799e6bc7e8959a8504c01d72f5e27a8e187fa53
Jul 07 18:16:32.878243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.878207    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958" event={"ID":"eb71b991-68f0-4548-929d-2cf2a3bdaa9f","Type":"ContainerStarted","Data":"befc096cb2c767a23d5eebc7a86ab45558a1456297ba3dcea2aa2377f429049b"}
Jul 07 18:16:32.878451 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.878249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958" event={"ID":"eb71b991-68f0-4548-929d-2cf2a3bdaa9f","Type":"ContainerStarted","Data":"af74ad97c835adfed8b848d5e799e6bc7e8959a8504c01d72f5e27a8e187fa53"}
Jul 07 18:16:32.892917 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:32.892867    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958" podStartSLOduration=1.892853984 podStartE2EDuration="1.892853984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:16:32.892260262 +0000 UTC m=+2154.466207616" watchObservedRunningTime="2026-07-07 18:16:32.892853984 +0000 UTC m=+2154.466801304"
Jul 07 18:16:35.025546 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:35.025502    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:16:42.204166 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:42.204102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:42.204556 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:42.204188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:42.216702 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:42.216673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:42.937795 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:42.937762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:16:49.985843 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:49.985810    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"]
Jul 07 18:16:49.988954 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:49.988933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:49.991456 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:49.991436    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"kube-root-ca.crt\""
Jul 07 18:16:49.992280 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:49.992263    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"openshift-service-ca.crt\""
Jul 07 18:16:49.992384 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:49.992299    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"default-dockercfg-5z65r\""
Jul 07 18:16:49.992384 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:49.992301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"auth-test-model-bc5a1a-kserve-self-signed-certs\""
Jul 07 18:16:50.000364 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.000342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"]
Jul 07 18:16:50.133531 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.133496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-home\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.133728 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.133543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7946a9-759d-4a40-a00f-019ebf025473-tls-certs\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.133728 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.133587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4lq\" (UniqueName: \"kubernetes.io/projected/2a7946a9-759d-4a40-a00f-019ebf025473-kube-api-access-gz4lq\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.133728 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.133644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-tmp-dir\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.133728 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.133706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-dshm\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.133940 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.133741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-model-cache\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.234644 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.234606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-tmp-dir\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.234831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.234654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-dshm\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.234831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.234678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-model-cache\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.234831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.234721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-home\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.234831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.234741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7946a9-759d-4a40-a00f-019ebf025473-tls-certs\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.234831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.234773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4lq\" (UniqueName: \"kubernetes.io/projected/2a7946a9-759d-4a40-a00f-019ebf025473-kube-api-access-gz4lq\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.235152 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.235095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-tmp-dir\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.235230 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.235149    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-model-cache\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.235230 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.235218    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-home\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.237118 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.237048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-dshm\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.237500 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.237484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7946a9-759d-4a40-a00f-019ebf025473-tls-certs\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.242965 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.242945    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4lq\" (UniqueName: \"kubernetes.io/projected/2a7946a9-759d-4a40-a00f-019ebf025473-kube-api-access-gz4lq\") pod \"auth-test-model-bc5a1a-kserve-cbb958855-hnqtk\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") " pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.301219 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.301182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:16:50.444431 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.444401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"]
Jul 07 18:16:50.450160 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:16:50.450127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7946a9_759d_4a40_a00f_019ebf025473.slice/crio-4195fb36883dfc34a28b404935bdf3bf48069d54258ada2a378946693fe5a47b WatchSource:0}: Error finding container 4195fb36883dfc34a28b404935bdf3bf48069d54258ada2a378946693fe5a47b: Status 404 returned error can't find the container with id 4195fb36883dfc34a28b404935bdf3bf48069d54258ada2a378946693fe5a47b
Jul 07 18:16:50.972624 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.972588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk" event={"ID":"2a7946a9-759d-4a40-a00f-019ebf025473","Type":"ContainerStarted","Data":"fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325"}
Jul 07 18:16:50.972793 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:50.972636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk" event={"ID":"2a7946a9-759d-4a40-a00f-019ebf025473","Type":"ContainerStarted","Data":"4195fb36883dfc34a28b404935bdf3bf48069d54258ada2a378946693fe5a47b"}
Jul 07 18:16:53.915692 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:53.915638    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk" podStartSLOduration=4.915619803 podStartE2EDuration="4.915619803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:16:50.988957844 +0000 UTC m=+2172.562905177" watchObservedRunningTime="2026-07-07 18:16:53.915619803 +0000 UTC m=+2175.489567143"
Jul 07 18:16:53.917265 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:16:53.917246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:17:00.301574 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:00.301539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:17:00.301974 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:00.301616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:17:00.313987 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:00.313963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:17:01.031696 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:01.031663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:17:10.701737 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:10.701698    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:17:26.902529 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:26.902493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:17:43.908666 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:17:43.908631    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:18:00.899390 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:00.899305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:18:16.700287 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:16.700251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:18:35.398605 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:35.398569    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:18:49.464691 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:49.464646    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"]
Jul 07 18:18:50.300593 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:50.300559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"]
Jul 07 18:18:53.107634 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:53.107599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:18:55.084916 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.084881    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"]
Jul 07 18:18:55.088305 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.088283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.090421 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.090395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a68ed4\"/\"sub-test-model-a68ed4-kserve-self-signed-certs\""
Jul 07 18:18:55.099093 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.099068    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"]
Jul 07 18:18:55.194306 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.194268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-dshm\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.194509 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.194472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-home\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.194587 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.194519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-tmp-dir\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.194587 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.194571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-model-cache\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.194698 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.194604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb69a44-6265-48c7-8603-565baaaa8c6c-tls-certs\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.194698 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.194649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqfh\" (UniqueName: \"kubernetes.io/projected/0cb69a44-6265-48c7-8603-565baaaa8c6c-kube-api-access-ktqfh\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.295646 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.295602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-dshm\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.295853 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.295703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-home\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.295853 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.295729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-tmp-dir\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.295853 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.295756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-model-cache\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.295853 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.295777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb69a44-6265-48c7-8603-565baaaa8c6c-tls-certs\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.295853 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.295840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqfh\" (UniqueName: \"kubernetes.io/projected/0cb69a44-6265-48c7-8603-565baaaa8c6c-kube-api-access-ktqfh\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.296157 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.296133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-home\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.296233 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.296189    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-tmp-dir\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.296233 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.296203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-model-cache\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.297926 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.297904    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-dshm\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.298241 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.298223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb69a44-6265-48c7-8603-565baaaa8c6c-tls-certs\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.303816 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.303793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqfh\" (UniqueName: \"kubernetes.io/projected/0cb69a44-6265-48c7-8603-565baaaa8c6c-kube-api-access-ktqfh\") pod \"sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") " pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.398807 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.398707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:18:55.537641 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.537608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"]
Jul 07 18:18:55.540871 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:18:55.540845    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb69a44_6265_48c7_8603_565baaaa8c6c.slice/crio-f4be31eb95a816c4fd327c8735e777ce736e50354b66f5e46df44db98d233cce WatchSource:0}: Error finding container f4be31eb95a816c4fd327c8735e777ce736e50354b66f5e46df44db98d233cce: Status 404 returned error can't find the container with id f4be31eb95a816c4fd327c8735e777ce736e50354b66f5e46df44db98d233cce
Jul 07 18:18:55.572971 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:55.572937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7" event={"ID":"0cb69a44-6265-48c7-8603-565baaaa8c6c","Type":"ContainerStarted","Data":"f4be31eb95a816c4fd327c8735e777ce736e50354b66f5e46df44db98d233cce"}
Jul 07 18:18:56.579858 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:56.579822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7" event={"ID":"0cb69a44-6265-48c7-8603-565baaaa8c6c","Type":"ContainerStarted","Data":"84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9"}
Jul 07 18:18:56.597185 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:18:56.597132    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7" podStartSLOduration=1.5970939450000001 podStartE2EDuration="1.597093945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:18:56.596043772 +0000 UTC m=+2298.169991312" watchObservedRunningTime="2026-07-07 18:18:56.597093945 +0000 UTC m=+2298.171041286"
Jul 07 18:19:04.485398 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.485348    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958" podUID="eb71b991-68f0-4548-929d-2cf2a3bdaa9f" containerName="main" containerID="cri-o://befc096cb2c767a23d5eebc7a86ab45558a1456297ba3dcea2aa2377f429049b" gracePeriod=45
Jul 07 18:19:04.619973 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.619936    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb71b991-68f0-4548-929d-2cf2a3bdaa9f" containerID="befc096cb2c767a23d5eebc7a86ab45558a1456297ba3dcea2aa2377f429049b" exitCode=0
Jul 07 18:19:04.620140 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.620003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958" event={"ID":"eb71b991-68f0-4548-929d-2cf2a3bdaa9f","Type":"ContainerDied","Data":"befc096cb2c767a23d5eebc7a86ab45558a1456297ba3dcea2aa2377f429049b"}
Jul 07 18:19:04.739741 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.739675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:19:04.775580 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmd9j\" (UniqueName: \"kubernetes.io/projected/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-kube-api-access-dmd9j\") pod \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") "
Jul 07 18:19:04.775580 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775584    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-dshm\") pod \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") "
Jul 07 18:19:04.775799 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tls-certs\") pod \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") "
Jul 07 18:19:04.775799 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775674    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tmp-dir\") pod \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") "
Jul 07 18:19:04.775799 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-model-cache\") pod \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") "
Jul 07 18:19:04.775799 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-home\") pod \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\" (UID: \"eb71b991-68f0-4548-929d-2cf2a3bdaa9f\") "
Jul 07 18:19:04.776032 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.775997    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb71b991-68f0-4548-929d-2cf2a3bdaa9f" (UID: "eb71b991-68f0-4548-929d-2cf2a3bdaa9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:04.776100 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.776087    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:04.776235 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.776197    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-home" (OuterVolumeSpecName: "home") pod "eb71b991-68f0-4548-929d-2cf2a3bdaa9f" (UID: "eb71b991-68f0-4548-929d-2cf2a3bdaa9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:04.776235 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.776224    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb71b991-68f0-4548-929d-2cf2a3bdaa9f" (UID: "eb71b991-68f0-4548-929d-2cf2a3bdaa9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:04.777934 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.777915    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-dshm" (OuterVolumeSpecName: "dshm") pod "eb71b991-68f0-4548-929d-2cf2a3bdaa9f" (UID: "eb71b991-68f0-4548-929d-2cf2a3bdaa9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:04.778173 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.778156    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb71b991-68f0-4548-929d-2cf2a3bdaa9f" (UID: "eb71b991-68f0-4548-929d-2cf2a3bdaa9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:19:04.778254 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.778184    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-kube-api-access-dmd9j" (OuterVolumeSpecName: "kube-api-access-dmd9j") pod "eb71b991-68f0-4548-929d-2cf2a3bdaa9f" (UID: "eb71b991-68f0-4548-929d-2cf2a3bdaa9f"). InnerVolumeSpecName "kube-api-access-dmd9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:19:04.876550 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.876516    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:04.876550 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.876544    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:04.876550 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.876554    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmd9j\" (UniqueName: \"kubernetes.io/projected/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-kube-api-access-dmd9j\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:04.876793 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.876563    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:04.876793 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:04.876574    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb71b991-68f0-4548-929d-2cf2a3bdaa9f-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.321499 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.321461    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk" podUID="2a7946a9-759d-4a40-a00f-019ebf025473" containerName="main" containerID="cri-o://fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325" gracePeriod=45
Jul 07 18:19:05.399495 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.399412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:19:05.399495 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.399474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:19:05.412046 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.412015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:19:05.589292 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.589268    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:19:05.625415 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.625387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958" event={"ID":"eb71b991-68f0-4548-929d-2cf2a3bdaa9f","Type":"ContainerDied","Data":"af74ad97c835adfed8b848d5e799e6bc7e8959a8504c01d72f5e27a8e187fa53"}
Jul 07 18:19:05.625415 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.625412    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"
Jul 07 18:19:05.625680 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.625441    2577 scope.go:117] "RemoveContainer" containerID="befc096cb2c767a23d5eebc7a86ab45558a1456297ba3dcea2aa2377f429049b"
Jul 07 18:19:05.626925 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.626893    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a7946a9-759d-4a40-a00f-019ebf025473" containerID="fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325" exitCode=0
Jul 07 18:19:05.627065 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.626936    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"
Jul 07 18:19:05.627065 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.627005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk" event={"ID":"2a7946a9-759d-4a40-a00f-019ebf025473","Type":"ContainerDied","Data":"fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325"}
Jul 07 18:19:05.627065 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.627041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk" event={"ID":"2a7946a9-759d-4a40-a00f-019ebf025473","Type":"ContainerDied","Data":"4195fb36883dfc34a28b404935bdf3bf48069d54258ada2a378946693fe5a47b"}
Jul 07 18:19:05.635839 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.635817    2577 scope.go:117] "RemoveContainer" containerID="fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325"
Jul 07 18:19:05.644346 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.644327    2577 scope.go:117] "RemoveContainer" containerID="fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325"
Jul 07 18:19:05.644561 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.644543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:19:05.644649 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:19:05.644626    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325\": container with ID starting with fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325 not found: ID does not exist" containerID="fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325"
Jul 07 18:19:05.644706 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.644664    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325"} err="failed to get container status \"fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325\": rpc error: code = NotFound desc = could not find container \"fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325\": container with ID starting with fe0c9611ccff824ea0aeb2307c30dd9ff844486c334907518f257055e4eb4325 not found: ID does not exist"
Jul 07 18:19:05.644706 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.644691    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"]
Jul 07 18:19:05.647511 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.647490    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/auth-test-model-a68ed4-kserve-6b4fbc6ff-h7958"]
Jul 07 18:19:05.683138 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-model-cache\") pod \"2a7946a9-759d-4a40-a00f-019ebf025473\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") "
Jul 07 18:19:05.683138 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683090    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-home\") pod \"2a7946a9-759d-4a40-a00f-019ebf025473\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") "
Jul 07 18:19:05.683338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7946a9-759d-4a40-a00f-019ebf025473-tls-certs\") pod \"2a7946a9-759d-4a40-a00f-019ebf025473\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") "
Jul 07 18:19:05.683338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz4lq\" (UniqueName: \"kubernetes.io/projected/2a7946a9-759d-4a40-a00f-019ebf025473-kube-api-access-gz4lq\") pod \"2a7946a9-759d-4a40-a00f-019ebf025473\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") "
Jul 07 18:19:05.683338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-dshm\") pod \"2a7946a9-759d-4a40-a00f-019ebf025473\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") "
Jul 07 18:19:05.683338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-tmp-dir\") pod \"2a7946a9-759d-4a40-a00f-019ebf025473\" (UID: \"2a7946a9-759d-4a40-a00f-019ebf025473\") "
Jul 07 18:19:05.683553 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683367    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a7946a9-759d-4a40-a00f-019ebf025473" (UID: "2a7946a9-759d-4a40-a00f-019ebf025473"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:05.683627 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683596    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.683627 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683603    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a7946a9-759d-4a40-a00f-019ebf025473" (UID: "2a7946a9-759d-4a40-a00f-019ebf025473"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:05.683856 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.683829    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-home" (OuterVolumeSpecName: "home") pod "2a7946a9-759d-4a40-a00f-019ebf025473" (UID: "2a7946a9-759d-4a40-a00f-019ebf025473"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:05.685933 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.685904    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-dshm" (OuterVolumeSpecName: "dshm") pod "2a7946a9-759d-4a40-a00f-019ebf025473" (UID: "2a7946a9-759d-4a40-a00f-019ebf025473"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:19:05.686026 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.685962    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7946a9-759d-4a40-a00f-019ebf025473-kube-api-access-gz4lq" (OuterVolumeSpecName: "kube-api-access-gz4lq") pod "2a7946a9-759d-4a40-a00f-019ebf025473" (UID: "2a7946a9-759d-4a40-a00f-019ebf025473"). InnerVolumeSpecName "kube-api-access-gz4lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:19:05.686026 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.685994    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7946a9-759d-4a40-a00f-019ebf025473-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a7946a9-759d-4a40-a00f-019ebf025473" (UID: "2a7946a9-759d-4a40-a00f-019ebf025473"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:19:05.784977 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.784948    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.784977 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.784979    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.784977 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.784992    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7946a9-759d-4a40-a00f-019ebf025473-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.785290 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.785004    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4lq\" (UniqueName: \"kubernetes.io/projected/2a7946a9-759d-4a40-a00f-019ebf025473-kube-api-access-gz4lq\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.785290 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.785016    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7946a9-759d-4a40-a00f-019ebf025473-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:19:05.948995 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.948936    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"]
Jul 07 18:19:05.951054 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:05.951032    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/auth-test-model-bc5a1a-kserve-cbb958855-hnqtk"]
Jul 07 18:19:07.096711 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:07.096680    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7946a9-759d-4a40-a00f-019ebf025473" path="/var/lib/kubelet/pods/2a7946a9-759d-4a40-a00f-019ebf025473/volumes"
Jul 07 18:19:07.097176 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:07.097045    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb71b991-68f0-4548-929d-2cf2a3bdaa9f" path="/var/lib/kubelet/pods/eb71b991-68f0-4548-929d-2cf2a3bdaa9f/volumes"
Jul 07 18:19:08.801399 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:08.801365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:19:42.696914 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.696879    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"]
Jul 07 18:19:42.698096 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.698069    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb71b991-68f0-4548-929d-2cf2a3bdaa9f" containerName="main"
Jul 07 18:19:42.698096 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.698097    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb71b991-68f0-4548-929d-2cf2a3bdaa9f" containerName="main"
Jul 07 18:19:42.698348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.698136    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7946a9-759d-4a40-a00f-019ebf025473" containerName="main"
Jul 07 18:19:42.698348 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.698146    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7946a9-759d-4a40-a00f-019ebf025473" containerName="main"
Jul 07 18:19:42.698483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.698364    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7946a9-759d-4a40-a00f-019ebf025473" containerName="main"
Jul 07 18:19:42.698483 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.698379    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb71b991-68f0-4548-929d-2cf2a3bdaa9f" containerName="main"
Jul 07 18:19:42.703702 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.703671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.707278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.707253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"kube-root-ca.crt\""
Jul 07 18:19:42.707419 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.707253    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"openshift-service-ca.crt\""
Jul 07 18:19:42.707419 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.707299    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"default-dockercfg-5z65r\""
Jul 07 18:19:42.707419 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.707358    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"sub-test-model-bc5a1a-kserve-self-signed-certs\""
Jul 07 18:19:42.709962 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.709937    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"]
Jul 07 18:19:42.721426 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.721402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-home\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.721530 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.721454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-model-cache\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.721530 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.721500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ece38b-3e12-46be-9972-fb32980d7783-tls-certs\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.721530 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.721518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bfk5\" (UniqueName: \"kubernetes.io/projected/85ece38b-3e12-46be-9972-fb32980d7783-kube-api-access-9bfk5\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.721687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.721537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-tmp-dir\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.721687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.721563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-dshm\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822239 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-model-cache\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822407 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ece38b-3e12-46be-9972-fb32980d7783-tls-certs\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822407 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfk5\" (UniqueName: \"kubernetes.io/projected/85ece38b-3e12-46be-9972-fb32980d7783-kube-api-access-9bfk5\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822407 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-tmp-dir\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822407 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-dshm\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822630 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-home\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822694 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822647    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-model-cache\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822752 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822708    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-tmp-dir\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.822811 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.822751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-home\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.824603 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.824581    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-dshm\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.824892 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.824876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ece38b-3e12-46be-9972-fb32980d7783-tls-certs\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:42.831441 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:42.831411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfk5\" (UniqueName: \"kubernetes.io/projected/85ece38b-3e12-46be-9972-fb32980d7783-kube-api-access-9bfk5\") pod \"sub-test-model-bc5a1a-kserve-6dc546865-w7wf4\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") " pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:43.014822 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:43.014787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:43.156165 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:43.156138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"]
Jul 07 18:19:43.158378 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:19:43.158355    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85ece38b_3e12_46be_9972_fb32980d7783.slice/crio-a068e35841e1252c29d15d5535ebf7fd691de913b4ab5c9531730270ade062b5 WatchSource:0}: Error finding container a068e35841e1252c29d15d5535ebf7fd691de913b4ab5c9531730270ade062b5: Status 404 returned error can't find the container with id a068e35841e1252c29d15d5535ebf7fd691de913b4ab5c9531730270ade062b5
Jul 07 18:19:43.810029 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:43.809989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4" event={"ID":"85ece38b-3e12-46be-9972-fb32980d7783","Type":"ContainerStarted","Data":"f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb"}
Jul 07 18:19:43.810029 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:43.810036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4" event={"ID":"85ece38b-3e12-46be-9972-fb32980d7783","Type":"ContainerStarted","Data":"a068e35841e1252c29d15d5535ebf7fd691de913b4ab5c9531730270ade062b5"}
Jul 07 18:19:43.826929 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:43.826878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4" podStartSLOduration=1.826862508 podStartE2EDuration="1.826862508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:19:43.825964262 +0000 UTC m=+2345.399911604" watchObservedRunningTime="2026-07-07 18:19:43.826862508 +0000 UTC m=+2345.400809885"
Jul 07 18:19:47.307814 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:47.307776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:19:53.015687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:53.015651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:53.015687 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:53.015697    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:53.030497 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:53.030468    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:19:53.873866 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:19:53.873836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:20:07.901479 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:07.901444    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:20:28.002778 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:28.002735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:20:32.536583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:32.536550    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"]
Jul 07 18:20:33.410599 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:33.410561    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"]
Jul 07 18:20:35.091191 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.091158    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"]
Jul 07 18:20:35.094828 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.094806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.097379 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.097360    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a68ed4\"/\"rate-test-model-a68ed4-kserve-self-signed-certs\""
Jul 07 18:20:35.104449 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.104418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"]
Jul 07 18:20:35.205970 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.205930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363cd13a-e911-4c36-8c66-95a5d3b31e92-tls-certs\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.205970 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.205975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-dshm\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.206205 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.205992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-model-cache\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.206205 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.206080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfs5\" (UniqueName: \"kubernetes.io/projected/363cd13a-e911-4c36-8c66-95a5d3b31e92-kube-api-access-7jfs5\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.206205 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.206148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-tmp-dir\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.206311 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.206214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-home\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.306663 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.306630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-tmp-dir\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.306833 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.306685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-home\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.306833 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.306718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363cd13a-e911-4c36-8c66-95a5d3b31e92-tls-certs\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.306833 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.306747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-dshm\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.306833 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.306770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-model-cache\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.306991 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.306847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfs5\" (UniqueName: \"kubernetes.io/projected/363cd13a-e911-4c36-8c66-95a5d3b31e92-kube-api-access-7jfs5\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.307166 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.307142    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-tmp-dir\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.307253 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.307185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-home\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.307313 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.307258    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-model-cache\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.309065 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.309038    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-dshm\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.309330 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.309312    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363cd13a-e911-4c36-8c66-95a5d3b31e92-tls-certs\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.314900 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.314878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfs5\" (UniqueName: \"kubernetes.io/projected/363cd13a-e911-4c36-8c66-95a5d3b31e92-kube-api-access-7jfs5\") pod \"rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") " pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.405525 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.405434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:35.537392 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:35.537361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"]
Jul 07 18:20:35.540516 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:20:35.540481    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod363cd13a_e911_4c36_8c66_95a5d3b31e92.slice/crio-237106804cd9a576a1413bba02a82e3b7b74ec66b57b3e7e39f4c4cec6f27551 WatchSource:0}: Error finding container 237106804cd9a576a1413bba02a82e3b7b74ec66b57b3e7e39f4c4cec6f27551: Status 404 returned error can't find the container with id 237106804cd9a576a1413bba02a82e3b7b74ec66b57b3e7e39f4c4cec6f27551
Jul 07 18:20:36.072837 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:36.072800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj" event={"ID":"363cd13a-e911-4c36-8c66-95a5d3b31e92","Type":"ContainerStarted","Data":"8e59f6d63db0b15cc521b2d115597977f4d7c8843639b7dd0ebd34a6c2088ee6"}
Jul 07 18:20:36.073009 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:36.072845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj" event={"ID":"363cd13a-e911-4c36-8c66-95a5d3b31e92","Type":"ContainerStarted","Data":"237106804cd9a576a1413bba02a82e3b7b74ec66b57b3e7e39f4c4cec6f27551"}
Jul 07 18:20:36.092340 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:36.092293    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj" podStartSLOduration=1.092278153 podStartE2EDuration="1.092278153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:20:36.091457529 +0000 UTC m=+2397.665404871" watchObservedRunningTime="2026-07-07 18:20:36.092278153 +0000 UTC m=+2397.666225494"
Jul 07 18:20:39.061567 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:39.061537    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:20:39.066295 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:39.066273    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bw4c2_c610d36a-d68f-4a47-90a0-62f695ccace9/ovn-acl-logging/0.log"
Jul 07 18:20:45.405543 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:45.405501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:45.405956 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:45.405560    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:45.418084 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:45.418058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:45.901589 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:45.901549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:20:46.134273 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:46.134245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:20:47.556773 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.556712    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7" podUID="0cb69a44-6265-48c7-8603-565baaaa8c6c" containerName="main" containerID="cri-o://84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9" gracePeriod=45
Jul 07 18:20:47.806415 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.806392    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:20:47.925346 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-model-cache\") pod \"0cb69a44-6265-48c7-8603-565baaaa8c6c\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") "
Jul 07 18:20:47.925346 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-dshm\") pod \"0cb69a44-6265-48c7-8603-565baaaa8c6c\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") "
Jul 07 18:20:47.925570 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-tmp-dir\") pod \"0cb69a44-6265-48c7-8603-565baaaa8c6c\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") "
Jul 07 18:20:47.925570 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-home\") pod \"0cb69a44-6265-48c7-8603-565baaaa8c6c\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") "
Jul 07 18:20:47.925570 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb69a44-6265-48c7-8603-565baaaa8c6c-tls-certs\") pod \"0cb69a44-6265-48c7-8603-565baaaa8c6c\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") "
Jul 07 18:20:47.925570 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktqfh\" (UniqueName: \"kubernetes.io/projected/0cb69a44-6265-48c7-8603-565baaaa8c6c-kube-api-access-ktqfh\") pod \"0cb69a44-6265-48c7-8603-565baaaa8c6c\" (UID: \"0cb69a44-6265-48c7-8603-565baaaa8c6c\") "
Jul 07 18:20:47.925570 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925502    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cb69a44-6265-48c7-8603-565baaaa8c6c" (UID: "0cb69a44-6265-48c7-8603-565baaaa8c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:47.925841 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925662    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-home" (OuterVolumeSpecName: "home") pod "0cb69a44-6265-48c7-8603-565baaaa8c6c" (UID: "0cb69a44-6265-48c7-8603-565baaaa8c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:47.925841 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925701    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cb69a44-6265-48c7-8603-565baaaa8c6c" (UID: "0cb69a44-6265-48c7-8603-565baaaa8c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:47.925841 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.925711    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:47.927624 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.927590    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb69a44-6265-48c7-8603-565baaaa8c6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cb69a44-6265-48c7-8603-565baaaa8c6c" (UID: "0cb69a44-6265-48c7-8603-565baaaa8c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:20:47.927624 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.927605    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-dshm" (OuterVolumeSpecName: "dshm") pod "0cb69a44-6265-48c7-8603-565baaaa8c6c" (UID: "0cb69a44-6265-48c7-8603-565baaaa8c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:47.927817 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:47.927621    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb69a44-6265-48c7-8603-565baaaa8c6c-kube-api-access-ktqfh" (OuterVolumeSpecName: "kube-api-access-ktqfh") pod "0cb69a44-6265-48c7-8603-565baaaa8c6c" (UID: "0cb69a44-6265-48c7-8603-565baaaa8c6c"). InnerVolumeSpecName "kube-api-access-ktqfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:20:48.026342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.026311    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.026342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.026337    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.026342 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.026346    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cb69a44-6265-48c7-8603-565baaaa8c6c-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.026565 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.026355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktqfh\" (UniqueName: \"kubernetes.io/projected/0cb69a44-6265-48c7-8603-565baaaa8c6c-kube-api-access-ktqfh\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.026565 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.026363    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cb69a44-6265-48c7-8603-565baaaa8c6c-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.134273 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.134241    2577 generic.go:356] "Generic (PLEG): container finished" podID="0cb69a44-6265-48c7-8603-565baaaa8c6c" containerID="84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9" exitCode=0
Jul 07 18:20:48.134448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.134287    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"
Jul 07 18:20:48.134448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.134314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7" event={"ID":"0cb69a44-6265-48c7-8603-565baaaa8c6c","Type":"ContainerDied","Data":"84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9"}
Jul 07 18:20:48.134448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.134357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7" event={"ID":"0cb69a44-6265-48c7-8603-565baaaa8c6c","Type":"ContainerDied","Data":"f4be31eb95a816c4fd327c8735e777ce736e50354b66f5e46df44db98d233cce"}
Jul 07 18:20:48.134448 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.134378    2577 scope.go:117] "RemoveContainer" containerID="84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9"
Jul 07 18:20:48.143466 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.143421    2577 scope.go:117] "RemoveContainer" containerID="84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9"
Jul 07 18:20:48.143711 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:20:48.143692    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9\": container with ID starting with 84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9 not found: ID does not exist" containerID="84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9"
Jul 07 18:20:48.143786 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.143719    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9"} err="failed to get container status \"84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9\": rpc error: code = NotFound desc = could not find container \"84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9\": container with ID starting with 84e5867d7c8609a84a7026088612138f18488995d182ddd8263f43416256dec9 not found: ID does not exist"
Jul 07 18:20:48.174045 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.174008    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"]
Jul 07 18:20:48.203006 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.202929    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/sub-test-model-a68ed4-kserve-5c4cfb799f-qnmc7"]
Jul 07 18:20:48.430438 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.430395    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4" podUID="85ece38b-3e12-46be-9972-fb32980d7783" containerName="main" containerID="cri-o://f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb" gracePeriod=45
Jul 07 18:20:48.684662 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.684637    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:20:48.732910 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.732877    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-tmp-dir\") pod \"85ece38b-3e12-46be-9972-fb32980d7783\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") "
Jul 07 18:20:48.733083 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.732921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bfk5\" (UniqueName: \"kubernetes.io/projected/85ece38b-3e12-46be-9972-fb32980d7783-kube-api-access-9bfk5\") pod \"85ece38b-3e12-46be-9972-fb32980d7783\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") "
Jul 07 18:20:48.733083 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.732977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-home\") pod \"85ece38b-3e12-46be-9972-fb32980d7783\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") "
Jul 07 18:20:48.733083 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-dshm\") pod \"85ece38b-3e12-46be-9972-fb32980d7783\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") "
Jul 07 18:20:48.733083 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-model-cache\") pod \"85ece38b-3e12-46be-9972-fb32980d7783\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") "
Jul 07 18:20:48.733083 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ece38b-3e12-46be-9972-fb32980d7783-tls-certs\") pod \"85ece38b-3e12-46be-9972-fb32980d7783\" (UID: \"85ece38b-3e12-46be-9972-fb32980d7783\") "
Jul 07 18:20:48.733380 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733222    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85ece38b-3e12-46be-9972-fb32980d7783" (UID: "85ece38b-3e12-46be-9972-fb32980d7783"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:48.733380 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733329    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-model-cache" (OuterVolumeSpecName: "model-cache") pod "85ece38b-3e12-46be-9972-fb32980d7783" (UID: "85ece38b-3e12-46be-9972-fb32980d7783"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:48.733380 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733340    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-home" (OuterVolumeSpecName: "home") pod "85ece38b-3e12-46be-9972-fb32980d7783" (UID: "85ece38b-3e12-46be-9972-fb32980d7783"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:48.733502 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733462    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.733502 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733482    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.733502 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.733498    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.735047 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.735010    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-dshm" (OuterVolumeSpecName: "dshm") pod "85ece38b-3e12-46be-9972-fb32980d7783" (UID: "85ece38b-3e12-46be-9972-fb32980d7783"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:20:48.735188 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.735122    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ece38b-3e12-46be-9972-fb32980d7783-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85ece38b-3e12-46be-9972-fb32980d7783" (UID: "85ece38b-3e12-46be-9972-fb32980d7783"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:20:48.735437 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.735242    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ece38b-3e12-46be-9972-fb32980d7783-kube-api-access-9bfk5" (OuterVolumeSpecName: "kube-api-access-9bfk5") pod "85ece38b-3e12-46be-9972-fb32980d7783" (UID: "85ece38b-3e12-46be-9972-fb32980d7783"). InnerVolumeSpecName "kube-api-access-9bfk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:20:48.834218 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.834141    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bfk5\" (UniqueName: \"kubernetes.io/projected/85ece38b-3e12-46be-9972-fb32980d7783-kube-api-access-9bfk5\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.834218 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.834165    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85ece38b-3e12-46be-9972-fb32980d7783-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:48.834218 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:48.834176    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ece38b-3e12-46be-9972-fb32980d7783-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:20:49.102339 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.098892    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb69a44-6265-48c7-8603-565baaaa8c6c" path="/var/lib/kubelet/pods/0cb69a44-6265-48c7-8603-565baaaa8c6c/volumes"
Jul 07 18:20:49.141072 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.141037    2577 generic.go:356] "Generic (PLEG): container finished" podID="85ece38b-3e12-46be-9972-fb32980d7783" containerID="f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb" exitCode=0
Jul 07 18:20:49.141249 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.141092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"
Jul 07 18:20:49.141249 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.141131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4" event={"ID":"85ece38b-3e12-46be-9972-fb32980d7783","Type":"ContainerDied","Data":"f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb"}
Jul 07 18:20:49.141249 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.141172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4" event={"ID":"85ece38b-3e12-46be-9972-fb32980d7783","Type":"ContainerDied","Data":"a068e35841e1252c29d15d5535ebf7fd691de913b4ab5c9531730270ade062b5"}
Jul 07 18:20:49.141249 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.141194    2577 scope.go:117] "RemoveContainer" containerID="f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb"
Jul 07 18:20:49.149801 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.149770    2577 scope.go:117] "RemoveContainer" containerID="f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb"
Jul 07 18:20:49.150049 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:20:49.150029    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb\": container with ID starting with f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb not found: ID does not exist" containerID="f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb"
Jul 07 18:20:49.150097 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.150060    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb"} err="failed to get container status \"f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb\": rpc error: code = NotFound desc = could not find container \"f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb\": container with ID starting with f0b30d0d03aeca63b2fd859f60319e83c5f56cf803a29b80feda0e8da97bbedb not found: ID does not exist"
Jul 07 18:20:49.181073 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.181045    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"]
Jul 07 18:20:49.189681 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:49.189654    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/sub-test-model-bc5a1a-kserve-6dc546865-w7wf4"]
Jul 07 18:20:51.096811 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:51.096774    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ece38b-3e12-46be-9972-fb32980d7783" path="/var/lib/kubelet/pods/85ece38b-3e12-46be-9972-fb32980d7783/volumes"
Jul 07 18:20:57.797372 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797337    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"]
Jul 07 18:20:57.797781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797687    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ece38b-3e12-46be-9972-fb32980d7783" containerName="main"
Jul 07 18:20:57.797781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797699    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ece38b-3e12-46be-9972-fb32980d7783" containerName="main"
Jul 07 18:20:57.797781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797716    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb69a44-6265-48c7-8603-565baaaa8c6c" containerName="main"
Jul 07 18:20:57.797781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797721    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb69a44-6265-48c7-8603-565baaaa8c6c" containerName="main"
Jul 07 18:20:57.797781 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797780    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ece38b-3e12-46be-9972-fb32980d7783" containerName="main"
Jul 07 18:20:57.797938 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.797789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb69a44-6265-48c7-8603-565baaaa8c6c" containerName="main"
Jul 07 18:20:57.802063 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.802037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:57.805259 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.805240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"kube-root-ca.crt\""
Jul 07 18:20:57.806043 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.806028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"rate-test-model-bc5a1a-kserve-self-signed-certs\""
Jul 07 18:20:57.806299 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.806283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"default-dockercfg-5z65r\""
Jul 07 18:20:57.806346 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.806283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc5a1a\"/\"openshift-service-ca.crt\""
Jul 07 18:20:57.811232 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.811213    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"]
Jul 07 18:20:57.927831 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.927791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlhd\" (UniqueName: \"kubernetes.io/projected/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-kube-api-access-8rlhd\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:57.927981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.927889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tmp-dir\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:57.927981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.927929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tls-certs\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:57.927981 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.927978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-dshm\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:57.928147 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.927999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-home\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:57.928147 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:57.928022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-model-cache\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.028549 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tmp-dir\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.028734 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tls-certs\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.028734 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-dshm\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.028734 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-home\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.028734 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-model-cache\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.028734 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlhd\" (UniqueName: \"kubernetes.io/projected/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-kube-api-access-8rlhd\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.029013 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.028966    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tmp-dir\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.029013 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.029002    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-home\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.029098 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.029041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-model-cache\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.031074 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.031049    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-dshm\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.031227 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.031081    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tls-certs\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.036388 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.036361    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlhd\" (UniqueName: \"kubernetes.io/projected/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-kube-api-access-8rlhd\") pod \"rate-test-model-bc5a1a-kserve-564c64d955-7cm4p\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") " pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.111583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.111499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:20:58.241804 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:58.241772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"]
Jul 07 18:20:58.245398 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:20:58.245363    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0774e87b_83e3_44a6_a5d6_5ef8275fe25b.slice/crio-e1635fb006f1fa56c2bae2f8e593b9ee79a71fd2406077e7dab09c4e898978dd WatchSource:0}: Error finding container e1635fb006f1fa56c2bae2f8e593b9ee79a71fd2406077e7dab09c4e898978dd: Status 404 returned error can't find the container with id e1635fb006f1fa56c2bae2f8e593b9ee79a71fd2406077e7dab09c4e898978dd
Jul 07 18:20:59.189566 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:59.189522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p" event={"ID":"0774e87b-83e3-44a6-a5d6-5ef8275fe25b","Type":"ContainerStarted","Data":"40b6590f536783062f18058e021f1ff9e4d1cf8605843de5b5429787acc7702f"}
Jul 07 18:20:59.189566 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:59.189565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p" event={"ID":"0774e87b-83e3-44a6-a5d6-5ef8275fe25b","Type":"ContainerStarted","Data":"e1635fb006f1fa56c2bae2f8e593b9ee79a71fd2406077e7dab09c4e898978dd"}
Jul 07 18:20:59.206426 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:20:59.206382    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p" podStartSLOduration=2.206366315 podStartE2EDuration="2.206366315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:20:59.204714525 +0000 UTC m=+2420.778662016" watchObservedRunningTime="2026-07-07 18:20:59.206366315 +0000 UTC m=+2420.780313656"
Jul 07 18:21:05.039198 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:05.039098    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:21:08.112677 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:08.112642    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:21:08.113177 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:08.112690    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:21:08.125245 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:08.125220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:21:08.243850 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:08.243820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:21:30.698327 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:30.698291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:21:45.097828 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:45.097801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:21:52.294087 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:52.294053    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"]
Jul 07 18:21:53.274586 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:53.274548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"]
Jul 07 18:21:58.897671 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:21:58.897633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:22:00.522226 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.522184    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7445-140227095888784"]
Jul 07 18:22:00.525749 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.525727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:00.527855 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.527831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8wxd5\""
Jul 07 18:22:00.530839 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.530815    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-140227095888784"]
Jul 07 18:22:00.672119 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.672077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnbt2\" (UniqueName: \"kubernetes.io/projected/06db1a9e-0c90-4d8f-89d4-cf8cead38998-kube-api-access-nnbt2\") pod \"test-curl-7445-140227095888784\" (UID: \"06db1a9e-0c90-4d8f-89d4-cf8cead38998\") " pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:00.773564 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.773482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnbt2\" (UniqueName: \"kubernetes.io/projected/06db1a9e-0c90-4d8f-89d4-cf8cead38998-kube-api-access-nnbt2\") pod \"test-curl-7445-140227095888784\" (UID: \"06db1a9e-0c90-4d8f-89d4-cf8cead38998\") " pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:00.781636 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.781607    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnbt2\" (UniqueName: \"kubernetes.io/projected/06db1a9e-0c90-4d8f-89d4-cf8cead38998-kube-api-access-nnbt2\") pod \"test-curl-7445-140227095888784\" (UID: \"06db1a9e-0c90-4d8f-89d4-cf8cead38998\") " pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:00.836055 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.836020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:00.969758 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.969730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-140227095888784"]
Jul 07 18:22:00.971908 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:22:00.971879    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06db1a9e_0c90_4d8f_89d4_cf8cead38998.slice/crio-4878e73c2efd94ccff42969e5d100301068364682920679ef68eb206dd6df1e8 WatchSource:0}: Error finding container 4878e73c2efd94ccff42969e5d100301068364682920679ef68eb206dd6df1e8: Status 404 returned error can't find the container with id 4878e73c2efd94ccff42969e5d100301068364682920679ef68eb206dd6df1e8
Jul 07 18:22:00.973142 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:00.973099    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 18:22:01.482198 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:01.482157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-140227095888784" event={"ID":"06db1a9e-0c90-4d8f-89d4-cf8cead38998","Type":"ContainerStarted","Data":"4878e73c2efd94ccff42969e5d100301068364682920679ef68eb206dd6df1e8"}
Jul 07 18:22:02.488349 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:02.488313    2577 generic.go:356] "Generic (PLEG): container finished" podID="06db1a9e-0c90-4d8f-89d4-cf8cead38998" containerID="6099bdd32e19af178a4957a69ae14ef840fcea6208abe339be85f12bce3fa8fe" exitCode=0
Jul 07 18:22:02.488748 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:02.488362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-140227095888784" event={"ID":"06db1a9e-0c90-4d8f-89d4-cf8cead38998","Type":"ContainerDied","Data":"6099bdd32e19af178a4957a69ae14ef840fcea6208abe339be85f12bce3fa8fe"}
Jul 07 18:22:03.629824 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.629798    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:03.664164 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.664135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7445-140227095888784_06db1a9e-0c90-4d8f-89d4-cf8cead38998/test-curl-7445-140227095888784/0.log"
Jul 07 18:22:03.689915 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.689881    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7445-140227095888784"]
Jul 07 18:22:03.693403 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.693380    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7445-140227095888784"]
Jul 07 18:22:03.701522 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.701438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnbt2\" (UniqueName: \"kubernetes.io/projected/06db1a9e-0c90-4d8f-89d4-cf8cead38998-kube-api-access-nnbt2\") pod \"06db1a9e-0c90-4d8f-89d4-cf8cead38998\" (UID: \"06db1a9e-0c90-4d8f-89d4-cf8cead38998\") "
Jul 07 18:22:03.703426 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.703401    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06db1a9e-0c90-4d8f-89d4-cf8cead38998-kube-api-access-nnbt2" (OuterVolumeSpecName: "kube-api-access-nnbt2") pod "06db1a9e-0c90-4d8f-89d4-cf8cead38998" (UID: "06db1a9e-0c90-4d8f-89d4-cf8cead38998"). InnerVolumeSpecName "kube-api-access-nnbt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:03.802884 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:03.802845    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnbt2\" (UniqueName: \"kubernetes.io/projected/06db1a9e-0c90-4d8f-89d4-cf8cead38998-kube-api-access-nnbt2\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:04.499677 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:04.499648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:04.499865 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:04.499698    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4878e73c2efd94ccff42969e5d100301068364682920679ef68eb206dd6df1e8"
Jul 07 18:22:04.509897 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:22:04.509873    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7445-140227095888784\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="06db1a9e-0c90-4d8f-89d4-cf8cead38998" pod="opendatahub/test-curl-7445-140227095888784"
Jul 07 18:22:05.096915 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:05.096870    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06db1a9e-0c90-4d8f-89d4-cf8cead38998" path="/var/lib/kubelet/pods/06db1a9e-0c90-4d8f-89d4-cf8cead38998/volumes"
Jul 07 18:22:07.316638 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.316548    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj" podUID="363cd13a-e911-4c36-8c66-95a5d3b31e92" containerName="main" containerID="cri-o://8e59f6d63db0b15cc521b2d115597977f4d7c8843639b7dd0ebd34a6c2088ee6" gracePeriod=45
Jul 07 18:22:07.516920 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.516890    2577 generic.go:356] "Generic (PLEG): container finished" podID="363cd13a-e911-4c36-8c66-95a5d3b31e92" containerID="8e59f6d63db0b15cc521b2d115597977f4d7c8843639b7dd0ebd34a6c2088ee6" exitCode=0
Jul 07 18:22:07.517067 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.516927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj" event={"ID":"363cd13a-e911-4c36-8c66-95a5d3b31e92","Type":"ContainerDied","Data":"8e59f6d63db0b15cc521b2d115597977f4d7c8843639b7dd0ebd34a6c2088ee6"}
Jul 07 18:22:07.577422 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.577366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:22:07.635903 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.635875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-home\") pod \"363cd13a-e911-4c36-8c66-95a5d3b31e92\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") "
Jul 07 18:22:07.636070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.635942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-model-cache\") pod \"363cd13a-e911-4c36-8c66-95a5d3b31e92\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") "
Jul 07 18:22:07.636070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.635973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jfs5\" (UniqueName: \"kubernetes.io/projected/363cd13a-e911-4c36-8c66-95a5d3b31e92-kube-api-access-7jfs5\") pod \"363cd13a-e911-4c36-8c66-95a5d3b31e92\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") "
Jul 07 18:22:07.636070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.635997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-tmp-dir\") pod \"363cd13a-e911-4c36-8c66-95a5d3b31e92\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") "
Jul 07 18:22:07.636070 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-dshm\") pod \"363cd13a-e911-4c36-8c66-95a5d3b31e92\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") "
Jul 07 18:22:07.636329 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363cd13a-e911-4c36-8c66-95a5d3b31e92-tls-certs\") pod \"363cd13a-e911-4c36-8c66-95a5d3b31e92\" (UID: \"363cd13a-e911-4c36-8c66-95a5d3b31e92\") "
Jul 07 18:22:07.636329 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636156    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-home" (OuterVolumeSpecName: "home") pod "363cd13a-e911-4c36-8c66-95a5d3b31e92" (UID: "363cd13a-e911-4c36-8c66-95a5d3b31e92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:07.636329 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636275    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-model-cache" (OuterVolumeSpecName: "model-cache") pod "363cd13a-e911-4c36-8c66-95a5d3b31e92" (UID: "363cd13a-e911-4c36-8c66-95a5d3b31e92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:07.636329 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636291    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "363cd13a-e911-4c36-8c66-95a5d3b31e92" (UID: "363cd13a-e911-4c36-8c66-95a5d3b31e92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:07.636517 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636427    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:07.636517 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636440    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:07.636517 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.636452    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:07.638186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.638161    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-dshm" (OuterVolumeSpecName: "dshm") pod "363cd13a-e911-4c36-8c66-95a5d3b31e92" (UID: "363cd13a-e911-4c36-8c66-95a5d3b31e92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:07.638266 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.638217    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363cd13a-e911-4c36-8c66-95a5d3b31e92-kube-api-access-7jfs5" (OuterVolumeSpecName: "kube-api-access-7jfs5") pod "363cd13a-e911-4c36-8c66-95a5d3b31e92" (UID: "363cd13a-e911-4c36-8c66-95a5d3b31e92"). InnerVolumeSpecName "kube-api-access-7jfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:07.638307 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.638262    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363cd13a-e911-4c36-8c66-95a5d3b31e92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "363cd13a-e911-4c36-8c66-95a5d3b31e92" (UID: "363cd13a-e911-4c36-8c66-95a5d3b31e92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:22:07.737125 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.737088    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfs5\" (UniqueName: \"kubernetes.io/projected/363cd13a-e911-4c36-8c66-95a5d3b31e92-kube-api-access-7jfs5\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:07.737291 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.737137    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/363cd13a-e911-4c36-8c66-95a5d3b31e92-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:07.737291 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:07.737147    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/363cd13a-e911-4c36-8c66-95a5d3b31e92-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:08.294611 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.294555    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p" podUID="0774e87b-83e3-44a6-a5d6-5ef8275fe25b" containerName="main" containerID="cri-o://40b6590f536783062f18058e021f1ff9e4d1cf8605843de5b5429787acc7702f" gracePeriod=45
Jul 07 18:22:08.524372 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.524336    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"
Jul 07 18:22:08.524819 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.524334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj" event={"ID":"363cd13a-e911-4c36-8c66-95a5d3b31e92","Type":"ContainerDied","Data":"237106804cd9a576a1413bba02a82e3b7b74ec66b57b3e7e39f4c4cec6f27551"}
Jul 07 18:22:08.524819 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.524450    2577 scope.go:117] "RemoveContainer" containerID="8e59f6d63db0b15cc521b2d115597977f4d7c8843639b7dd0ebd34a6c2088ee6"
Jul 07 18:22:08.526268 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.526240    2577 generic.go:356] "Generic (PLEG): container finished" podID="0774e87b-83e3-44a6-a5d6-5ef8275fe25b" containerID="40b6590f536783062f18058e021f1ff9e4d1cf8605843de5b5429787acc7702f" exitCode=0
Jul 07 18:22:08.526414 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.526270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p" event={"ID":"0774e87b-83e3-44a6-a5d6-5ef8275fe25b","Type":"ContainerDied","Data":"40b6590f536783062f18058e021f1ff9e4d1cf8605843de5b5429787acc7702f"}
Jul 07 18:22:08.555064 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.555012    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:22:08.560923 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.560901    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"]
Jul 07 18:22:08.566715 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.566696    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a68ed4/rate-test-model-a68ed4-kserve-6c9ccff56c-hrftj"]
Jul 07 18:22:08.645457 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-dshm\") pod \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") "
Jul 07 18:22:08.645457 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-home\") pod \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") "
Jul 07 18:22:08.645681 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645485    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rlhd\" (UniqueName: \"kubernetes.io/projected/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-kube-api-access-8rlhd\") pod \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") "
Jul 07 18:22:08.645681 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tmp-dir\") pod \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") "
Jul 07 18:22:08.645681 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tls-certs\") pod \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") "
Jul 07 18:22:08.645681 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-model-cache\") pod \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\" (UID: \"0774e87b-83e3-44a6-a5d6-5ef8275fe25b\") "
Jul 07 18:22:08.645893 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645752    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-home" (OuterVolumeSpecName: "home") pod "0774e87b-83e3-44a6-a5d6-5ef8275fe25b" (UID: "0774e87b-83e3-44a6-a5d6-5ef8275fe25b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:08.645893 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645823    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0774e87b-83e3-44a6-a5d6-5ef8275fe25b" (UID: "0774e87b-83e3-44a6-a5d6-5ef8275fe25b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:08.645986 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645919    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0774e87b-83e3-44a6-a5d6-5ef8275fe25b" (UID: "0774e87b-83e3-44a6-a5d6-5ef8275fe25b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:08.645986 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645934    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-home\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:08.645986 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.645955    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tmp-dir\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:08.647644 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.647622    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-dshm" (OuterVolumeSpecName: "dshm") pod "0774e87b-83e3-44a6-a5d6-5ef8275fe25b" (UID: "0774e87b-83e3-44a6-a5d6-5ef8275fe25b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:08.647734 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.647650    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0774e87b-83e3-44a6-a5d6-5ef8275fe25b" (UID: "0774e87b-83e3-44a6-a5d6-5ef8275fe25b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 18:22:08.647769 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.647757    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-kube-api-access-8rlhd" (OuterVolumeSpecName: "kube-api-access-8rlhd") pod "0774e87b-83e3-44a6-a5d6-5ef8275fe25b" (UID: "0774e87b-83e3-44a6-a5d6-5ef8275fe25b"). InnerVolumeSpecName "kube-api-access-8rlhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:08.746516 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.746481    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-tls-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:08.746516 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.746510    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-model-cache\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:08.746516 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.746521    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-dshm\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:08.746755 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:08.746532    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rlhd\" (UniqueName: \"kubernetes.io/projected/0774e87b-83e3-44a6-a5d6-5ef8275fe25b-kube-api-access-8rlhd\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:09.097679 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.097645    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="363cd13a-e911-4c36-8c66-95a5d3b31e92" path="/var/lib/kubelet/pods/363cd13a-e911-4c36-8c66-95a5d3b31e92/volumes"
Jul 07 18:22:09.533379 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.533351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"
Jul 07 18:22:09.533798 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.533352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p" event={"ID":"0774e87b-83e3-44a6-a5d6-5ef8275fe25b","Type":"ContainerDied","Data":"e1635fb006f1fa56c2bae2f8e593b9ee79a71fd2406077e7dab09c4e898978dd"}
Jul 07 18:22:09.533798 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.533486    2577 scope.go:117] "RemoveContainer" containerID="40b6590f536783062f18058e021f1ff9e4d1cf8605843de5b5429787acc7702f"
Jul 07 18:22:09.539859 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.539833    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7445-140227095889504"]
Jul 07 18:22:09.540256 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540240    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0774e87b-83e3-44a6-a5d6-5ef8275fe25b" containerName="main"
Jul 07 18:22:09.540338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540258    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0774e87b-83e3-44a6-a5d6-5ef8275fe25b" containerName="main"
Jul 07 18:22:09.540338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540289    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06db1a9e-0c90-4d8f-89d4-cf8cead38998" containerName="test-curl-7445-140227095888784"
Jul 07 18:22:09.540338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540300    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="06db1a9e-0c90-4d8f-89d4-cf8cead38998" containerName="test-curl-7445-140227095888784"
Jul 07 18:22:09.540338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540318    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="363cd13a-e911-4c36-8c66-95a5d3b31e92" containerName="main"
Jul 07 18:22:09.540338 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540326    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="363cd13a-e911-4c36-8c66-95a5d3b31e92" containerName="main"
Jul 07 18:22:09.540644 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540421    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0774e87b-83e3-44a6-a5d6-5ef8275fe25b" containerName="main"
Jul 07 18:22:09.540644 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540435    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="06db1a9e-0c90-4d8f-89d4-cf8cead38998" containerName="test-curl-7445-140227095888784"
Jul 07 18:22:09.540644 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.540445    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="363cd13a-e911-4c36-8c66-95a5d3b31e92" containerName="main"
Jul 07 18:22:09.544764 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.544745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:09.547256 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.547239    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8wxd5\""
Jul 07 18:22:09.548820 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.548796    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"]
Jul 07 18:22:09.551181 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.551153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-140227095889504"]
Jul 07 18:22:09.553473 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.553449    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc5a1a/rate-test-model-bc5a1a-kserve-564c64d955-7cm4p"]
Jul 07 18:22:09.654375 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.654343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrxp\" (UniqueName: \"kubernetes.io/projected/fbea57d6-5e5a-4171-9738-f7fd5f90c07c-kube-api-access-zwrxp\") pod \"test-curl-7445-140227095889504\" (UID: \"fbea57d6-5e5a-4171-9738-f7fd5f90c07c\") " pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:09.755761 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.755726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrxp\" (UniqueName: \"kubernetes.io/projected/fbea57d6-5e5a-4171-9738-f7fd5f90c07c-kube-api-access-zwrxp\") pod \"test-curl-7445-140227095889504\" (UID: \"fbea57d6-5e5a-4171-9738-f7fd5f90c07c\") " pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:09.763870 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.763841    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrxp\" (UniqueName: \"kubernetes.io/projected/fbea57d6-5e5a-4171-9738-f7fd5f90c07c-kube-api-access-zwrxp\") pod \"test-curl-7445-140227095889504\" (UID: \"fbea57d6-5e5a-4171-9738-f7fd5f90c07c\") " pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:09.855496 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.855419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:09.985384 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:09.985352    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7445-140227095889504"]
Jul 07 18:22:09.988234 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:22:09.988203    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbea57d6_5e5a_4171_9738_f7fd5f90c07c.slice/crio-d875314c3f403101f80389cc85677ae4f35d7db625361e67f8dd2a27fabf9309 WatchSource:0}: Error finding container d875314c3f403101f80389cc85677ae4f35d7db625361e67f8dd2a27fabf9309: Status 404 returned error can't find the container with id d875314c3f403101f80389cc85677ae4f35d7db625361e67f8dd2a27fabf9309
Jul 07 18:22:10.539670 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:10.539588    2577 generic.go:356] "Generic (PLEG): container finished" podID="fbea57d6-5e5a-4171-9738-f7fd5f90c07c" containerID="d9cfc969d92533a165f3da0e7ca17ff7fef1e1f7f08766e6b2c385f2cb09c8e9" exitCode=0
Jul 07 18:22:10.540162 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:10.539675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-140227095889504" event={"ID":"fbea57d6-5e5a-4171-9738-f7fd5f90c07c","Type":"ContainerDied","Data":"d9cfc969d92533a165f3da0e7ca17ff7fef1e1f7f08766e6b2c385f2cb09c8e9"}
Jul 07 18:22:10.540162 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:10.539715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7445-140227095889504" event={"ID":"fbea57d6-5e5a-4171-9738-f7fd5f90c07c","Type":"ContainerStarted","Data":"d875314c3f403101f80389cc85677ae4f35d7db625361e67f8dd2a27fabf9309"}
Jul 07 18:22:11.096932 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.096894    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0774e87b-83e3-44a6-a5d6-5ef8275fe25b" path="/var/lib/kubelet/pods/0774e87b-83e3-44a6-a5d6-5ef8275fe25b/volumes"
Jul 07 18:22:11.675271 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.675244    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:11.698793 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.698768    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7445-140227095889504_fbea57d6-5e5a-4171-9738-f7fd5f90c07c/test-curl-7445-140227095889504/0.log"
Jul 07 18:22:11.724255 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.724226    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7445-140227095889504"]
Jul 07 18:22:11.726139 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.726103    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7445-140227095889504"]
Jul 07 18:22:11.774018 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.773989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwrxp\" (UniqueName: \"kubernetes.io/projected/fbea57d6-5e5a-4171-9738-f7fd5f90c07c-kube-api-access-zwrxp\") pod \"fbea57d6-5e5a-4171-9738-f7fd5f90c07c\" (UID: \"fbea57d6-5e5a-4171-9738-f7fd5f90c07c\") "
Jul 07 18:22:11.776094 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.776073    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbea57d6-5e5a-4171-9738-f7fd5f90c07c-kube-api-access-zwrxp" (OuterVolumeSpecName: "kube-api-access-zwrxp") pod "fbea57d6-5e5a-4171-9738-f7fd5f90c07c" (UID: "fbea57d6-5e5a-4171-9738-f7fd5f90c07c"). InnerVolumeSpecName "kube-api-access-zwrxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:11.875536 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:11.875449    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwrxp\" (UniqueName: \"kubernetes.io/projected/fbea57d6-5e5a-4171-9738-f7fd5f90c07c-kube-api-access-zwrxp\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:12.551126 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:12.551091    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:12.551304 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:12.551145    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d875314c3f403101f80389cc85677ae4f35d7db625361e67f8dd2a27fabf9309"
Jul 07 18:22:12.561026 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:22:12.560999    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7445-140227095889504\" is forbidden: User \"system:node:ip-10-0-138-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-144.ec2.internal' and this object" podUID="fbea57d6-5e5a-4171-9738-f7fd5f90c07c" pod="opendatahub/test-curl-7445-140227095889504"
Jul 07 18:22:13.097281 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:13.097248    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbea57d6-5e5a-4171-9738-f7fd5f90c07c" path="/var/lib/kubelet/pods/fbea57d6-5e5a-4171-9738-f7fd5f90c07c/volumes"
Jul 07 18:22:16.311439 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:16.311404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:22:18.945722 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:18.945685    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"]
Jul 07 18:22:18.946178 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:18.945918    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" containerID="cri-o://7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23" gracePeriod=30
Jul 07 18:22:23.262741 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:23.262700    2577 patch_prober.go:28] interesting pod/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 18:22:23.263164 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:23.262772    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 18:22:24.201125 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.201083    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:22:24.294230 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ddc40-752d-4246-bc71-f1eb441d19e1-istiod-ca-cert\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294255    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-token\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-podinfo\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngzv2\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-kube-api-access-ngzv2\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-socket\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-credential-socket\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-data\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-certs\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.294665 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294496    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-envoy\") pod \"b47ddc40-752d-4246-bc71-f1eb441d19e1\" (UID: \"b47ddc40-752d-4246-bc71-f1eb441d19e1\") "
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294661    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294705    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294727    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47ddc40-752d-4246-bc71-f1eb441d19e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294835    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ddc40-752d-4246-bc71-f1eb441d19e1-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294852    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294865    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.295186 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.294900    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:24.295551 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.295364    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:24.296700 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.296677    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:24.296700 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.296688    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-kube-api-access-ngzv2" (OuterVolumeSpecName: "kube-api-access-ngzv2") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "kube-api-access-ngzv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:24.296930 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.296909    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:24.296989 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.296905    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b47ddc40-752d-4246-bc71-f1eb441d19e1" (UID: "b47ddc40-752d-4246-bc71-f1eb441d19e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:22:24.396155 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.396093    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.396155 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.396151    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.396155 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.396162    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.396406 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.396170    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.396406 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.396178    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ddc40-752d-4246-bc71-f1eb441d19e1-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.396406 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.396186    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzv2\" (UniqueName: \"kubernetes.io/projected/b47ddc40-752d-4246-bc71-f1eb441d19e1-kube-api-access-ngzv2\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:24.612183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.612074    2577 generic.go:356] "Generic (PLEG): container finished" podID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerID="7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23" exitCode=0
Jul 07 18:22:24.612183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.612145    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"
Jul 07 18:22:24.612183 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.612150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" event={"ID":"b47ddc40-752d-4246-bc71-f1eb441d19e1","Type":"ContainerDied","Data":"7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23"}
Jul 07 18:22:24.612459 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.612209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9" event={"ID":"b47ddc40-752d-4246-bc71-f1eb441d19e1","Type":"ContainerDied","Data":"241cd57a4a57495f3e60a7db6e656690767bc5a24422c2c1bb5c86593258dda7"}
Jul 07 18:22:24.612459 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.612231    2577 scope.go:117] "RemoveContainer" containerID="7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23"
Jul 07 18:22:24.621410 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.621394    2577 scope.go:117] "RemoveContainer" containerID="2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"
Jul 07 18:22:24.629117 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.629082    2577 scope.go:117] "RemoveContainer" containerID="7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23"
Jul 07 18:22:24.629361 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:22:24.629343    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23\": container with ID starting with 7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23 not found: ID does not exist" containerID="7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23"
Jul 07 18:22:24.629422 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.629370    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23"} err="failed to get container status \"7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23\": rpc error: code = NotFound desc = could not find container \"7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23\": container with ID starting with 7433fb79d5471b1c14fb84a028d0e509cc0e628a055ec4bd4b60cc085df4cf23 not found: ID does not exist"
Jul 07 18:22:24.629422 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.629388    2577 scope.go:117] "RemoveContainer" containerID="2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"
Jul 07 18:22:24.629627 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:22:24.629610    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025\": container with ID starting with 2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025 not found: ID does not exist" containerID="2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"
Jul 07 18:22:24.629684 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.629638    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025"} err="failed to get container status \"2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025\": rpc error: code = NotFound desc = could not find container \"2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025\": container with ID starting with 2f765b71ec1f7e76a9622386982fdb0b826dcaff74bb7571d3eaa329cec87025 not found: ID does not exist"
Jul 07 18:22:24.635639 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.635618    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"]
Jul 07 18:22:24.639796 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:24.639776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a68ed4-openshift-default-698c9fb5cf-78lv9"]
Jul 07 18:22:25.097168 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:25.097130    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" path="/var/lib/kubelet/pods/b47ddc40-752d-4246-bc71-f1eb441d19e1/volumes"
Jul 07 18:22:25.956347 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:25.955499    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"]
Jul 07 18:22:25.956347 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:25.955764    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy" containerID="cri-o://7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336" gracePeriod=30
Jul 07 18:22:29.294387 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:29.294351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-btrpx"]
Jul 07 18:22:31.224091 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.224032    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:22:31.362426 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-podinfo\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362426 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-certs\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-credential-socket\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxlz\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-kube-api-access-6hxlz\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-socket\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-data\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istiod-ca-cert\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.362693 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-token\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.363020 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-envoy\") pod \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\" (UID: \"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5\") "
Jul 07 18:22:31.363020 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362719    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:31.363020 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.362792    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:31.363203 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.363020    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-certs\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.363203 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.363039    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-credential-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.363203 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.363062    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:31.363203 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.363080    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 18:22:31.363556 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.363527    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-data" (OuterVolumeSpecName: "istio-data") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:31.364804 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.364773    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 18:22:31.364896 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.364840    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 18:22:31.364896 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.364854    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-kube-api-access-6hxlz" (OuterVolumeSpecName: "kube-api-access-6hxlz") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "kube-api-access-6hxlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:31.364896 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.364873    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-token" (OuterVolumeSpecName: "istio-token") pod "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" (UID: "6172ed2a-a67e-49bb-8bed-f60bec1b1ea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 18:22:31.463594 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463557    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-envoy\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.463594 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463587    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-podinfo\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.463594 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463596    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxlz\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-kube-api-access-6hxlz\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.463840 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463605    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-workload-socket\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.463840 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463616    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-data\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.463840 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463624    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istiod-ca-cert\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.463840 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.463632    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5-istio-token\") on node \"ip-10-0-138-144.ec2.internal\" DevicePath \"\""
Jul 07 18:22:31.650604 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.650568    2577 generic.go:356] "Generic (PLEG): container finished" podID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerID="7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336" exitCode=0
Jul 07 18:22:31.650810 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.650623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" event={"ID":"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5","Type":"ContainerDied","Data":"7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336"}
Jul 07 18:22:31.650810 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.650629    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"
Jul 07 18:22:31.650810 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.650667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll" event={"ID":"6172ed2a-a67e-49bb-8bed-f60bec1b1ea5","Type":"ContainerDied","Data":"9b95fbbb2b397e42764cf368c68632a6e5f35704d197b4fcd36cf9faf2fe3467"}
Jul 07 18:22:31.650810 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.650688    2577 scope.go:117] "RemoveContainer" containerID="7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336"
Jul 07 18:22:31.659966 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.659948    2577 scope.go:117] "RemoveContainer" containerID="79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"
Jul 07 18:22:31.667583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.667564    2577 scope.go:117] "RemoveContainer" containerID="7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336"
Jul 07 18:22:31.667849 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:22:31.667830    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336\": container with ID starting with 7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336 not found: ID does not exist" containerID="7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336"
Jul 07 18:22:31.667929 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.667861    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336"} err="failed to get container status \"7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336\": rpc error: code = NotFound desc = could not find container \"7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336\": container with ID starting with 7e7a71703a3284246f0a461fe1ead2929afe055437cc458359ecb337ddd80336 not found: ID does not exist"
Jul 07 18:22:31.667929 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.667885    2577 scope.go:117] "RemoveContainer" containerID="79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"
Jul 07 18:22:31.668157 ip-10-0-138-144 kubenswrapper[2577]: E0707 18:22:31.668139    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9\": container with ID starting with 79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9 not found: ID does not exist" containerID="79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"
Jul 07 18:22:31.668207 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.668163    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9"} err="failed to get container status \"79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9\": rpc error: code = NotFound desc = could not find container \"79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9\": container with ID starting with 79b9570c7fc03d0fdc79a7cc687d58266dc3fd3983e29d11a2899c4c78416bb9 not found: ID does not exist"
Jul 07 18:22:31.672837 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.672814    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"]
Jul 07 18:22:31.679979 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.679956    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bc5a1a-openshift-default-5b5fbccc59-894ll"]
Jul 07 18:22:31.765140 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.765088    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5cgnt_2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9/manager/0.log"
Jul 07 18:22:31.912317 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:31.912205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zlfbr_84c02517-5eb7-4070-bd06-a9f0cd5f99ce/manager/0.log"
Jul 07 18:22:32.167493 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:32.167425    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724120-kw48k_b6c127e6-05d7-4ab5-8982-9023281d28dc/cleanup/0.log"
Jul 07 18:22:32.548139 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:32.548100    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pzgtv_f88f0257-449e-4759-b23d-f9f1d36acce7/server/0.log"
Jul 07 18:22:32.692940 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:32.692906    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lbzc6_4e99d36f-1252-4e68-9eab-2d6a40e88192/manager/0.log"
Jul 07 18:22:32.825049 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:32.824978    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f77d785b9-5hwvh_032c7bb8-1e8a-4ec3-a667-cd81e01053d8/copy-manifests/0.log"
Jul 07 18:22:32.832679 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:32.832643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f77d785b9-5hwvh_032c7bb8-1e8a-4ec3-a667-cd81e01053d8/manager/0.log"
Jul 07 18:22:33.097026 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:33.096946    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" path="/var/lib/kubelet/pods/6172ed2a-a67e-49bb-8bed-f60bec1b1ea5/volumes"
Jul 07 18:22:34.875593 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:34.875560    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-975k5_6b67ddc1-340e-4c64-aa94-5a6ad183421c/manager/0.log"
Jul 07 18:22:34.993974 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:34.993945    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bq267_08bdeede-a044-48b2-a90d-ed7a9fa6bf18/manager/0.log"
Jul 07 18:22:35.243702 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:35.243619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-q2mk9_c8622673-b8f5-4cc6-bf73-a2139b185195/registry-server/0.log"
Jul 07 18:22:35.530609 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:35.530576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-btrpx_aff70cc8-445d-4a9e-bed3-1ca0fce32221/limitador/0.log"
Jul 07 18:22:35.679802 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:35.679754    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-2jhtr_e788a6ab-bd40-4f77-a0b1-e2ffaf032f94/manager/0.log"
Jul 07 18:22:36.091384 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:36.091336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rq928_e081b8b6-ee75-44a4-b5b3-796ca0309b99/istio-proxy/0.log"
Jul 07 18:22:36.349278 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:36.349194    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5675bf5659-c5zcw_51fba43c-8cfe-4daf-bf0c-9841ca500aec/kube-auth-proxy/0.log"
Jul 07 18:22:36.617287 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:36.617187    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/1.log"
Jul 07 18:22:37.906898 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:37.906868    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jwbxw_69b58f8a-15a7-411e-8e0d-9cf7f548355e/main/0.log"
Jul 07 18:22:42.503767 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.503731    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6gdp2/must-gather-88s4g"]
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504063    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504074    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504088    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504097    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504102    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504127    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbea57d6-5e5a-4171-9738-f7fd5f90c07c" containerName="test-curl-7445-140227095889504"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504133    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbea57d6-5e5a-4171-9738-f7fd5f90c07c" containerName="test-curl-7445-140227095889504"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504186    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504193    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47ddc40-752d-4246-bc71-f1eb441d19e1" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504200    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbea57d6-5e5a-4171-9738-f7fd5f90c07c" containerName="test-curl-7445-140227095889504"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504205    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy"
Jul 07 18:22:42.504243 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy"
Jul 07 18:22:42.504651 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504270    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy"
Jul 07 18:22:42.504651 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.504275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6172ed2a-a67e-49bb-8bed-f60bec1b1ea5" containerName="istio-proxy"
Jul 07 18:22:42.508762 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.508745    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.512117 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.512082    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6gdp2\"/\"kube-root-ca.crt\""
Jul 07 18:22:42.512240 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.512193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6gdp2\"/\"openshift-service-ca.crt\""
Jul 07 18:22:42.512240 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.512218    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6gdp2\"/\"default-dockercfg-scf4p\""
Jul 07 18:22:42.523577 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.523553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6gdp2/must-gather-88s4g"]
Jul 07 18:22:42.661678 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.661640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e45683cb-2c2c-4692-b441-1498ca750f33-must-gather-output\") pod \"must-gather-88s4g\" (UID: \"e45683cb-2c2c-4692-b441-1498ca750f33\") " pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.661881 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.661706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwm74\" (UniqueName: \"kubernetes.io/projected/e45683cb-2c2c-4692-b441-1498ca750f33-kube-api-access-cwm74\") pod \"must-gather-88s4g\" (UID: \"e45683cb-2c2c-4692-b441-1498ca750f33\") " pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.762440 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.762341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e45683cb-2c2c-4692-b441-1498ca750f33-must-gather-output\") pod \"must-gather-88s4g\" (UID: \"e45683cb-2c2c-4692-b441-1498ca750f33\") " pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.762440 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.762400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwm74\" (UniqueName: \"kubernetes.io/projected/e45683cb-2c2c-4692-b441-1498ca750f33-kube-api-access-cwm74\") pod \"must-gather-88s4g\" (UID: \"e45683cb-2c2c-4692-b441-1498ca750f33\") " pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.762703 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.762679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e45683cb-2c2c-4692-b441-1498ca750f33-must-gather-output\") pod \"must-gather-88s4g\" (UID: \"e45683cb-2c2c-4692-b441-1498ca750f33\") " pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.770626 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.770594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwm74\" (UniqueName: \"kubernetes.io/projected/e45683cb-2c2c-4692-b441-1498ca750f33-kube-api-access-cwm74\") pod \"must-gather-88s4g\" (UID: \"e45683cb-2c2c-4692-b441-1498ca750f33\") " pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.818090 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.818065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6gdp2/must-gather-88s4g"
Jul 07 18:22:42.952159 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:42.952132    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6gdp2/must-gather-88s4g"]
Jul 07 18:22:42.954575 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:22:42.954546    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode45683cb_2c2c_4692_b441_1498ca750f33.slice/crio-5758da660f72042a34e25d93bee69bf2f427fc2fb38ef6ccfc504095cfe7f6ba WatchSource:0}: Error finding container 5758da660f72042a34e25d93bee69bf2f427fc2fb38ef6ccfc504095cfe7f6ba: Status 404 returned error can't find the container with id 5758da660f72042a34e25d93bee69bf2f427fc2fb38ef6ccfc504095cfe7f6ba
Jul 07 18:22:43.710355 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:43.710320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6gdp2/must-gather-88s4g" event={"ID":"e45683cb-2c2c-4692-b441-1498ca750f33","Type":"ContainerStarted","Data":"5758da660f72042a34e25d93bee69bf2f427fc2fb38ef6ccfc504095cfe7f6ba"}
Jul 07 18:22:44.717868 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:44.717834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6gdp2/must-gather-88s4g" event={"ID":"e45683cb-2c2c-4692-b441-1498ca750f33","Type":"ContainerStarted","Data":"3f4b65d66f9aa0393731fde7a82d0052d9e8e9e1fd50278a60565f2c4269c56f"}
Jul 07 18:22:44.718367 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:44.717877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6gdp2/must-gather-88s4g" event={"ID":"e45683cb-2c2c-4692-b441-1498ca750f33","Type":"ContainerStarted","Data":"6e1afe0e4bca2e1aea916cc214bb45943d1a39100a2ef54f58fc170111bd0b42"}
Jul 07 18:22:44.733089 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:44.733021    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6gdp2/must-gather-88s4g" podStartSLOduration=1.656999634 podStartE2EDuration="2.733000563s" totalImagesPullingTime="1.076000929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:22:44.732102639 +0000 UTC m=+2526.306049993" watchObservedRunningTime="2026-07-07 18:22:44.733000563 +0000 UTC m=+2526.306947905"
Jul 07 18:22:45.622621 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:45.622593    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4rd2v_1b3c6dc9-7a2e-4f5c-8568-003c50ee0a51/global-pull-secret-syncer/0.log"
Jul 07 18:22:45.852880 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:45.852826    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-glpm8_d1d094db-aab0-4daa-8ac6-39e13bd8d3ee/konnectivity-agent/0.log"
Jul 07 18:22:45.939791 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:45.939688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-144.ec2.internal_d33f28cff3cad66dfe5aa1e747b9f44c/haproxy/0.log"
Jul 07 18:22:50.773468 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:50.773429    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-975k5_6b67ddc1-340e-4c64-aa94-5a6ad183421c/manager/0.log"
Jul 07 18:22:50.803553 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:50.803520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bq267_08bdeede-a044-48b2-a90d-ed7a9fa6bf18/manager/0.log"
Jul 07 18:22:50.872755 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:50.872718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-q2mk9_c8622673-b8f5-4cc6-bf73-a2139b185195/registry-server/0.log"
Jul 07 18:22:51.015401 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:51.015369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-btrpx_aff70cc8-445d-4a9e-bed3-1ca0fce32221/limitador/0.log"
Jul 07 18:22:51.092663 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:51.092579    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-2jhtr_e788a6ab-bd40-4f77-a0b1-e2ffaf032f94/manager/0.log"
Jul 07 18:22:53.428872 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:53.428834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkpch_b7d4032b-4c92-460a-b54f-4c142e2244cc/node-exporter/0.log"
Jul 07 18:22:53.452957 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:53.452924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkpch_b7d4032b-4c92-460a-b54f-4c142e2244cc/kube-rbac-proxy/0.log"
Jul 07 18:22:53.475611 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:53.475580    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tkpch_b7d4032b-4c92-460a-b54f-4c142e2244cc/init-textfile/0.log"
Jul 07 18:22:55.208057 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:55.208025    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-gkbmq_a44a5ab6-5c81-4e25-8e12-4e78eaa1624a/networking-console-plugin/0.log"
Jul 07 18:22:55.923773 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:55.923734    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"]
Jul 07 18:22:55.930004 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:55.929977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:55.936904 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:55.936870    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"]
Jul 07 18:22:56.005565 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.005525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qsb\" (UniqueName: \"kubernetes.io/projected/7c801a62-3ea7-4bb7-a85c-78867c2909b7-kube-api-access-r5qsb\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.005843 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.005821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-lib-modules\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.005961 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.005948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-proc\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.006072 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.006061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-sys\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.006290 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.006269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-podres\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107379 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-podres\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107589 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qsb\" (UniqueName: \"kubernetes.io/projected/7c801a62-3ea7-4bb7-a85c-78867c2909b7-kube-api-access-r5qsb\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107589 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-lib-modules\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107589 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-proc\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107589 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-sys\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107589 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107568    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-podres\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107638    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-sys\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-proc\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.107867 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.107697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c801a62-3ea7-4bb7-a85c-78867c2909b7-lib-modules\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.116293 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.116253    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qsb\" (UniqueName: \"kubernetes.io/projected/7c801a62-3ea7-4bb7-a85c-78867c2909b7-kube-api-access-r5qsb\") pod \"perf-node-gather-daemonset-bkjt8\" (UID: \"7c801a62-3ea7-4bb7-a85c-78867c2909b7\") " pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.244506 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.244417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.430807 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.429845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"]
Jul 07 18:22:56.434231 ip-10-0-138-144 kubenswrapper[2577]: W0707 18:22:56.434184    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c801a62_3ea7_4bb7_a85c_78867c2909b7.slice/crio-ca7ea50315168c9402acf0bd96dd98e5c3161133d7c137cdf0ec3ff37bf2fe46 WatchSource:0}: Error finding container ca7ea50315168c9402acf0bd96dd98e5c3161133d7c137cdf0ec3ff37bf2fe46: Status 404 returned error can't find the container with id ca7ea50315168c9402acf0bd96dd98e5c3161133d7c137cdf0ec3ff37bf2fe46
Jul 07 18:22:56.802395 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.802312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8" event={"ID":"7c801a62-3ea7-4bb7-a85c-78867c2909b7","Type":"ContainerStarted","Data":"8c5a07a20aca2f599ea27702cf925a6ea95d501691d6412500ffe832ab7efc97"}
Jul 07 18:22:56.802395 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.802364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8" event={"ID":"7c801a62-3ea7-4bb7-a85c-78867c2909b7","Type":"ContainerStarted","Data":"ca7ea50315168c9402acf0bd96dd98e5c3161133d7c137cdf0ec3ff37bf2fe46"}
Jul 07 18:22:56.802583 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.802448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:22:56.819335 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:56.819278    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8" podStartSLOduration=1.8192574160000001 podStartE2EDuration="1.819257416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:22:56.818455162 +0000 UTC m=+2538.392402503" watchObservedRunningTime="2026-07-07 18:22:56.819257416 +0000 UTC m=+2538.393204761"
Jul 07 18:22:58.052128 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:58.052074    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bpht_5e624548-2110-4f89-b179-c30ea2dd9c08/dns/0.log"
Jul 07 18:22:58.072978 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:58.072948    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bpht_5e624548-2110-4f89-b179-c30ea2dd9c08/kube-rbac-proxy/0.log"
Jul 07 18:22:58.196031 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:58.196006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fv45b_0e039226-8e03-42d0-b084-f552e7b4c2b5/dns-node-resolver/0.log"
Jul 07 18:22:58.790612 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:58.790558    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79768b874c-tskt2_7ef7e83f-6f3a-4a43-934e-b45dc796aeda/registry/0.log"
Jul 07 18:22:58.813547 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:58.813517    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kntg_d2e560e2-d355-45c9-90ab-7c6baca64f5b/node-ca/0.log"
Jul 07 18:22:59.769826 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:59.769799    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rq928_e081b8b6-ee75-44a4-b5b3-796ca0309b99/istio-proxy/0.log"
Jul 07 18:22:59.906887 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:59.906851    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5675bf5659-c5zcw_51fba43c-8cfe-4daf-bf0c-9841ca500aec/kube-auth-proxy/0.log"
Jul 07 18:22:59.961816 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:22:59.961788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/0.log"
Jul 07 18:23:00.019144 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:00.019095    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/1.log"
Jul 07 18:23:00.858953 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:00.858892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h4xxw_15b03e63-d408-4d95-a9e5-60b317778fc3/serve-healthcheck-canary/0.log"
Jul 07 18:23:01.377763 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:01.377734    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2s8zn_60c44b63-0ee5-4f79-94c3-b2538bfb6088/kube-rbac-proxy/0.log"
Jul 07 18:23:01.401896 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:01.401868    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2s8zn_60c44b63-0ee5-4f79-94c3-b2538bfb6088/exporter/0.log"
Jul 07 18:23:01.429879 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:01.429809    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2s8zn_60c44b63-0ee5-4f79-94c3-b2538bfb6088/extractor/0.log"
Jul 07 18:23:02.819743 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:02.819702    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6gdp2/perf-node-gather-daemonset-bkjt8"
Jul 07 18:23:03.670529 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:03.670496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5cgnt_2a8bbc4f-0ffd-434f-a887-b0f9dc3ad7a9/manager/0.log"
Jul 07 18:23:03.916649 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:03.916614    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zlfbr_84c02517-5eb7-4070-bd06-a9f0cd5f99ce/manager/0.log"
Jul 07 18:23:03.975488 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:03.975405    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724120-kw48k_b6c127e6-05d7-4ab5-8982-9023281d28dc/cleanup/0.log"
Jul 07 18:23:04.130508 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:04.130478    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-pzgtv_f88f0257-449e-4759-b23d-f9f1d36acce7/server/0.log"
Jul 07 18:23:04.236723 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:04.236636    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lbzc6_4e99d36f-1252-4e68-9eab-2d6a40e88192/manager/0.log"
Jul 07 18:23:04.264416 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:04.264379    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f77d785b9-5hwvh_032c7bb8-1e8a-4ec3-a667-cd81e01053d8/manager/0.log"
Jul 07 18:23:04.287642 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:04.287617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f77d785b9-5hwvh_032c7bb8-1e8a-4ec3-a667-cd81e01053d8/copy-manifests/0.log"
Jul 07 18:23:06.035953 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:06.035923    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qc9lm_77c4489e-d8e3-460f-ae46-ab6f2e098fa6/openshift-lws-operator/0.log"
Jul 07 18:23:06.406559 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:06.406530    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rq928_e081b8b6-ee75-44a4-b5b3-796ca0309b99/istio-proxy/0.log"
Jul 07 18:23:06.513486 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:06.513457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5675bf5659-c5zcw_51fba43c-8cfe-4daf-bf0c-9841ca500aec/kube-auth-proxy/0.log"
Jul 07 18:23:06.550018 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:06.549990    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/0.log"
Jul 07 18:23:06.599126 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:06.599085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/1.log"
Jul 07 18:23:07.909690 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:07.909660    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rq928_e081b8b6-ee75-44a4-b5b3-796ca0309b99/istio-proxy/0.log"
Jul 07 18:23:08.022780 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:08.022750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5675bf5659-c5zcw_51fba43c-8cfe-4daf-bf0c-9841ca500aec/kube-auth-proxy/0.log"
Jul 07 18:23:08.059235 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:08.059203    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/0.log"
Jul 07 18:23:08.109515 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:08.109434    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-stz9d_77e8b1fd-f9f6-41ea-aba7-b893af5fc41f/istio-proxy/1.log"
Jul 07 18:23:11.234356 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:11.234322    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4t562_946bb130-666d-4fb0-a145-af56ee5d2e85/migrator/0.log"
Jul 07 18:23:11.256069 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:11.256039    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-4t562_946bb130-666d-4fb0-a145-af56ee5d2e85/graceful-termination/0.log"
Jul 07 18:23:12.650391 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.650366    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/kube-multus-additional-cni-plugins/0.log"
Jul 07 18:23:12.675574 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.675547    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/egress-router-binary-copy/0.log"
Jul 07 18:23:12.695877 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.695853    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/cni-plugins/0.log"
Jul 07 18:23:12.717122 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.717085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/bond-cni-plugin/0.log"
Jul 07 18:23:12.739202 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.739169    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/routeoverride-cni/0.log"
Jul 07 18:23:12.760547 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.760520    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/whereabouts-cni-bincopy/0.log"
Jul 07 18:23:12.782622 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:12.782599    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pd4lj_988895c8-9fa3-482b-97a1-9815c60b8afa/whereabouts-cni/0.log"
Jul 07 18:23:13.164020 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:13.163986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dzkll_0ad89e8b-8da5-44a0-aedc-fe8a67ca0ccf/kube-multus/0.log"
Jul 07 18:23:13.188684 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:13.188654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6s6cr_e6523d94-f705-49fb-ac51-83169974230c/network-metrics-daemon/0.log"
Jul 07 18:23:13.205037 ip-10-0-138-144 kubenswrapper[2577]: I0707 18:23:13.205010    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6s6cr_e6523d94-f705-49fb-ac51-83169974230c/kube-rbac-proxy/0.log"